Jul 27 21:45:21.136800 ip-10-0-129-80 systemd[1]: Starting Kubernetes Kubelet... Jul 27 21:45:21.505211 ip-10-0-129-80 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 21:45:21.505211 ip-10-0-129-80 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 21:45:21.505211 ip-10-0-129-80 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 21:45:21.505211 ip-10-0-129-80 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 21:45:21.505211 ip-10-0-129-80 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 21:45:21.508275 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.508157 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 21:45:21.512417 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512402 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:45:21.512417 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512417 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512420 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512428 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512432 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512435 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512438 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512441 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512444 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512448 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512451 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512454 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512457 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512461 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512466 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:45:21.512488 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512470 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512489 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512508 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512512 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512515 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512518 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512520 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512523 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512525 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512528 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512531 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512534 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512536 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512539 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512541 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512544 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512546 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512549 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512553 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:45:21.512822 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512557 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512561 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512565 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512567 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512571 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512575 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512578 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512580 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512583 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512587 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512590 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512592 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512594 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512597 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512600 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512603 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512606 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512609 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512611 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512614 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:45:21.513527 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512617 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512619 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512622 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512625 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512628 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512630 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512633 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512635 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512637 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512640 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512643 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512646 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512648 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512650 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512655 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512659 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512662 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512666 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512669 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:45:21.514174 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512671 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512675 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512678 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512680 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512683 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512685 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512688 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512691 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512694 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512697 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512699 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512702 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512704 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512707 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512709 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512712 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512715 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512718 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512720 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512723 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:45:21.514660 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512725 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:45:21.515152 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512728 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:45:21.515152 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512730 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:45:21.515152 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512733 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:45:21.515152 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512735 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:45:21.515152 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.512738 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:45:21.515438 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515423 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:45:21.515438 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515438 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515442 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515445 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515448 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515452 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515455 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515457 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515461 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515464 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515466 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515469 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515472 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515490 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515494 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515497 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515499 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515502 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515505 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515507 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515510 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:45:21.515504 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515513 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515516 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515519 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515521 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515524 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515527 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515530 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515532 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515535 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515538 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515540 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515543 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515546 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515548 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515551 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515554 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515556 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515559 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515562 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515564 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:45:21.516006 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515567 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515569 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515571 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515574 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515577 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515579 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515582 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515586 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515590 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515594 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515596 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515599 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515601 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515604 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515607 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515610 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515613 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515615 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515618 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515621 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:45:21.516494 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515623 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515627 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515629 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515631 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515634 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515637 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515639 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515642 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515646 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515649 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515652 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515655 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515658 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515660 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515663 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515667 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515671 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515674 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515676 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:45:21.516986 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515680 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515683 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515685 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515688 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515691 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515694 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515697 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515700 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515703 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515706 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515709 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515711 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515714 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515717 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515726 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515729 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515732 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515734 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.515737 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:45:21.517449 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516138 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516144 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516147 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516150 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516153 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516155 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516158 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516161 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516163 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516167 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516171 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516174 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516178 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516180 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516183 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516186 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516189 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516192 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516194 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516197 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:45:21.517922 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516201 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516203 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516206 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516208 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516211 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516214 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516216 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516219 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516223 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516227 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516230 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516233 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516236 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516238 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516241 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516243 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516246 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516248 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516251 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:45:21.518396 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516253 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516256 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516258 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516261 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516263 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516266 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516269 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516271 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516275 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516277 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516281 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516283 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516286 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516289 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516292 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516294 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516296 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516299 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516301 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:45:21.518872 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516304 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516306 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516309 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516311 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516314 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516318 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516321 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516324 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516327 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516329 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516332 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516335 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516337 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516340 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516342 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516345 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516347 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516349 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516352 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516355 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:45:21.519367 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516357 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516360 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516362 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516365 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516368 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516370 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516373 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516376 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516379 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516381 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516384 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516386 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516389 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516392 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516394 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516397 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516399 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516401 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516404 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516406 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:45:21.519871 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.516409 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516492 2561 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516500 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516507 2561 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516512 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516516 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516519 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516523 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516527 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516530 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516534 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516538 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516542 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516545 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516548 2561 flags.go:64] FLAG: --cgroup-root="" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516551 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516555 2561 flags.go:64] FLAG: --client-ca-file="" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516559 2561 flags.go:64] FLAG: --cloud-provider="external" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516562 2561 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516566 2561 flags.go:64] FLAG: --cluster-domain="" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516569 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516572 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516576 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516579 2561 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 21:45:21.520348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516587 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516590 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516593 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516597 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516600 2561 flags.go:64] FLAG: --contention-profiling="false" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516603 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516606 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516609 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516612 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516617 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516619 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516623 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516626 2561 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516629 2561 flags.go:64] FLAG: --enable-server="true" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516632 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516636 2561 flags.go:64] FLAG: --event-burst="100" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516639 2561 flags.go:64] FLAG: --event-qps="50" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516642 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516645 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516648 2561 flags.go:64] FLAG: --eviction-hard="" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516652 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516655 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516658 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516661 2561 flags.go:64] FLAG: --eviction-soft="" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516664 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 21:45:21.520932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516667 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516671 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516674 2561 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516677 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516679 2561 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516683 2561 flags.go:64] FLAG: --feature-gates="" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516686 2561 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516689 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516692 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516695 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516698 2561 flags.go:64] FLAG: --healthz-port="10248" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516701 2561 flags.go:64] FLAG: --help="false" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516704 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-129-80.ec2.internal" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516707 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516710 2561 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516713 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516716 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516719 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516722 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516725 2561 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516728 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516730 2561 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516733 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516736 2561 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 21:45:21.521580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516740 2561 flags.go:64] FLAG: --kube-reserved="" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516742 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516745 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516748 2561 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516752 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516755 2561 flags.go:64] FLAG: --lock-file="" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516757 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516760 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516763 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516768 2561 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516772 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516775 2561 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516778 2561 flags.go:64] FLAG: --logging-format="text" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516780 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516784 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516787 2561 flags.go:64] FLAG: --manifest-url="" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516789 2561 flags.go:64] FLAG: --manifest-url-header="" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516796 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516799 2561 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516803 2561 flags.go:64] FLAG: --max-pods="110" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516807 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516809 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516813 2561 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516815 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516818 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 21:45:21.522168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516822 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516824 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516832 2561 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516835 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516838 2561 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516841 2561 flags.go:64] FLAG: --pod-cidr="" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516844 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516850 2561 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516852 2561 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516855 2561 flags.go:64] FLAG: --pods-per-core="0" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516858 2561 flags.go:64] FLAG: --port="10250" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516861 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516866 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d24b6a8657824bbc" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516870 2561 flags.go:64] FLAG: --qos-reserved="" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516872 2561 flags.go:64] FLAG: --read-only-port="10255" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516875 2561 flags.go:64] FLAG: --register-node="true" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516878 2561 flags.go:64] FLAG: --register-with-taints="" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516882 2561 flags.go:64] FLAG: --registry-burst="10" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516885 2561 flags.go:64] FLAG: --registry-qps="5" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516888 2561 flags.go:64] FLAG: --reserved-cpus="" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516891 2561 flags.go:64] FLAG: --reserved-memory="" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516894 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516897 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516900 2561 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516904 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 21:45:21.522871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516907 2561 flags.go:64] FLAG: --runonce="false" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516910 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516913 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516916 2561 flags.go:64] FLAG: --seccomp-default="false" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516919 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516921 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516924 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516927 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516930 2561 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516933 2561 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516936 2561 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516939 2561 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516942 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516945 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516948 2561 flags.go:64] FLAG: --system-cgroups="" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516951 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516956 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516958 2561 flags.go:64] FLAG: --tls-cert-file="" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516961 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516966 2561 flags.go:64] FLAG: --tls-min-version="" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516969 2561 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516972 2561 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516975 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516977 2561 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516980 2561 flags.go:64] FLAG: --v="2" Jul 27 21:45:21.523527 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516985 2561 flags.go:64] FLAG: --version="false" Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516989 2561 flags.go:64] FLAG: --vmodule="" Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516993 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.516996 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517113 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517117 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517121 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517124 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517127 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517130 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517133 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517135 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517138 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517140 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517143 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517145 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517148 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517151 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517153 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517156 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517158 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:45:21.524142 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517161 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517163 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517166 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517168 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517170 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517174 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517177 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517180 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517183 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517185 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517187 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517190 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517193 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517196 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517198 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517201 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517203 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517208 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517211 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:45:21.524657 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517214 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517217 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517220 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517222 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517225 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517227 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517230 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517232 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517235 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517237 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517240 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517243 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517245 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517248 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517250 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517253 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517270 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517274 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517278 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517281 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:45:21.525126 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517284 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517286 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517289 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517292 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517295 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517297 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517300 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517303 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517306 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517309 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517313 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517316 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517318 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517321 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517323 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517326 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517328 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517331 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517334 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517336 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:45:21.525629 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517339 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517342 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517345 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517347 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517350 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517353 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517355 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517357 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517360 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517362 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517367 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517370 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517372 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517375 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517377 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517381 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517384 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517388 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517392 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:45:21.526117 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517395 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517398 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517401 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.517405 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.518055 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.524507 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.524522 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.525284 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.525291 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525354 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525360 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525363 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525366 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525370 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:45:21.526591 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525373 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525376 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525379 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525381 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525384 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525387 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525390 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525392 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525395 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525398 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525400 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525403 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525405 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525408 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525410 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525413 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525416 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525418 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525421 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525424 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:45:21.526942 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525427 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525430 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525432 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525435 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525438 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525441 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525444 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525446 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525449 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525451 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525454 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525457 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525459 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525462 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525464 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525467 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525469 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525472 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525488 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525491 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:45:21.527422 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525494 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525497 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525500 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525502 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525505 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525508 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525510 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525513 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525516 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525518 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525520 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525523 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525544 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525547 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525551 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525554 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525557 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525560 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525563 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:45:21.527915 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525565 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525568 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525571 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525574 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525576 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525579 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525581 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525584 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525587 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525589 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525592 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525595 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525599 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525602 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525605 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525608 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525612 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525615 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525619 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:45:21.528467 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525622 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525624 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525627 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525630 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525633 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525636 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525639 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525642 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525646 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525649 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525652 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525655 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525659 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525662 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525665 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:45:21.528945 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525668 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.525674 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525794 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525799 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525802 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525805 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525808 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525810 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525813 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525816 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525818 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525821 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525824 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525827 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525829 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:45:21.529305 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525832 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525835 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525837 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525840 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525843 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525845 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525848 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525850 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525853 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525855 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525859 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525861 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525864 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525866 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525869 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525872 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525874 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525877 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525879 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525881 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:45:21.529711 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525884 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525886 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525889 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525891 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525894 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525896 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525898 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525901 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525903 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525906 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525909 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525913 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525916 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525918 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525921 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525923 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525926 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525929 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525931 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:45:21.530199 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525935 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525938 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525941 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525944 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525947 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525949 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525952 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525955 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525957 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525960 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525962 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525966 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525969 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525972 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525975 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525978 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525980 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525983 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525986 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525988 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:45:21.530676 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525991 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525993 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525996 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.525998 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526000 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526003 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526005 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526008 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526010 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526013 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526015 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526017 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526020 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526022 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526025 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526028 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526030 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526033 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526036 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526039 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:45:21.531193 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526041 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:45:21.531836 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526044 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:45:21.531836 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526046 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:45:21.531836 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526048 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:45:21.531836 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526051 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:45:21.531836 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526053 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:45:21.531836 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:21.526056 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:45:21.531836 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.526061 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 21:45:21.531836 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.526179 2561 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 21:45:21.531836 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.528119 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 21:45:21.531836 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.529324 2561 server.go:1019] "Starting client certificate rotation" Jul 27 21:45:21.531836 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.529421 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 21:45:21.531836 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.529457 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 21:45:21.531836 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.531716 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 21:45:21.535149 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.535131 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 21:45:21.542906 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.542881 2561 log.go:25] "Validated CRI v1 runtime API" Jul 27 21:45:21.544994 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.544979 2561 log.go:25] "Validated CRI v1 image API" Jul 27 21:45:21.545751 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.545735 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 21:45:21.548172 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.548152 2561 fs.go:135] Filesystem UUIDs: map[23a79023-98a9-4e9b-89ea-393086864faf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d22b4364-db07-4e08-8413-92defd81217a:/dev/nvme0n1p4] Jul 27 21:45:21.548230 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.548172 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 21:45:21.553207 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.553102 2561 manager.go:216] Machine: {Timestamp:2026-07-27 21:45:21.551917151 +0000 UTC m=+0.302137555 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103164 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e8844381f42ca1723482869cc403d SystemUUID:ec2e8844-381f-42ca-1723-482869cc403d BootID:2bc4339f-2ab6-442f-89f9-2d5dbd9100cc Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7f:bf:d6:73:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7f:bf:d6:73:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:05:e4:05:a7:81 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 21:45:21.553207 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.553205 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 21:45:21.553333 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.553321 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 21:45:21.556488 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.556453 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 21:45:21.556620 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.556493 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 21:45:21.557678 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.557668 2561 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 21:45:21.557715 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.557681 2561 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 21:45:21.557715 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.557689 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 21:45:21.557900 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.557886 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 21:45:21.558337 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.558324 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 21:45:21.559089 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.559079 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 21:45:21.560736 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.560724 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 27 21:45:21.560865 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.560856 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 21:45:21.560896 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.560886 2561 kubelet.go:486] "Attempting to sync node with API server" Jul 27 21:45:21.560924 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.560903 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 21:45:21.560924 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.560920 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 21:45:21.560978 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.560929 2561 kubelet.go:398] "Adding apiserver pod source" Jul 27 21:45:21.560978 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.560939 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 21:45:21.561296 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.561285 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 21:45:21.561340 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.561304 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 21:45:21.563584 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.563567 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 21:45:21.565529 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.565516 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 21:45:21.565566 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.565546 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 21:45:21.567064 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567052 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 21:45:21.567106 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567068 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 21:45:21.567106 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567074 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 21:45:21.567106 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567080 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 21:45:21.567106 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567089 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 21:45:21.567106 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567094 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 21:45:21.567106 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567103 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 21:45:21.567106 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567108 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 21:45:21.567278 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567115 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 21:45:21.567278 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567121 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 21:45:21.567278 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567137 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 21:45:21.567278 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567143 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 21:45:21.567742 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567729 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 21:45:21.567776 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.567744 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 21:45:21.570090 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.570071 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 21:45:21.570396 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.570381 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-80.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 21:45:21.570450 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.570437 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 21:45:21.570927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.570914 2561 server.go:1268] "Started kubelet" Jul 27 21:45:21.571069 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.571003 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 21:45:21.571171 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.571046 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 21:45:21.571171 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.571161 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 21:45:21.571738 ip-10-0-129-80 systemd[1]: Started Kubernetes Kubelet. Jul 27 21:45:21.572746 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.572688 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 21:45:21.572935 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.572921 2561 server.go:310] "Adding debug handlers to kubelet server" Jul 27 21:45:21.578392 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.578372 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 21:45:21.580506 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.580490 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 21:45:21.580594 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.580517 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 21:45:21.580594 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.580586 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 27 21:45:21.581062 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.581042 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 21:45:21.581144 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.581070 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 21:45:21.581144 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.581047 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 21:45:21.581238 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.581206 2561 reconstruct.go:97] "Volume reconstruction finished" Jul 27 21:45:21.581238 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.581212 2561 reconciler.go:29] "Reconciler: start to sync state" Jul 27 21:45:21.581331 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.581258 2561 factory.go:55] Registering systemd factory Jul 27 21:45:21.581331 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.581275 2561 factory.go:223] Registration of the systemd container factory successfully Jul 27 21:45:21.581499 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.581462 2561 factory.go:153] Registering CRI-O factory Jul 27 21:45:21.581499 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.581494 2561 factory.go:223] Registration of the crio container factory successfully Jul 27 21:45:21.581620 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.581547 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 21:45:21.581620 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.581571 2561 factory.go:103] Registering Raw factory Jul 27 21:45:21.581620 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.581584 2561 manager.go:1196] Started watching for new ooms in manager Jul 27 21:45:21.581917 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.581903 2561 manager.go:349] Starting recovery of all containers Jul 27 21:45:21.583610 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.583580 2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 27 21:45:21.583733 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.583709 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 27 21:45:21.584454 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.583195 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-80.ec2.internal.18c643af16442eb1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-80.ec2.internal,UID:ip-10-0-129-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-80.ec2.internal,},FirstTimestamp:2026-07-27 21:45:21.570885297 +0000 UTC m=+0.321105700,LastTimestamp:2026-07-27 21:45:21.570885297 +0000 UTC m=+0.321105700,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-80.ec2.internal,}" Jul 27 21:45:21.589663 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.589612 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 21:45:21.591540 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.591523 2561 manager.go:354] Recovery completed Jul 27 21:45:21.592768 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.592738 2561 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 27 21:45:21.595913 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.595900 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:45:21.597759 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.597728 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:45:21.597841 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.597773 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:45:21.597841 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.597784 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:45:21.598370 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.598345 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 21:45:21.598370 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.598360 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 21:45:21.598370 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.598358 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:45:21.598534 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.598377 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 27 21:45:21.599033 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.599011 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:45:21.599085 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.599039 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:45:21.599085 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.599051 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:45:21.599903 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.599892 2561 policy_none.go:49] "None policy: Start" Jul 27 21:45:21.599958 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.599907 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 21:45:21.599958 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.599917 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 21:45:21.600876 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.600861 2561 policy_none.go:47] "Start" Jul 27 21:45:21.605911 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.605840 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-80.ec2.internal.18c643af17de348c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-80.ec2.internal,UID:ip-10-0-129-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-80.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-80.ec2.internal,},FirstTimestamp:2026-07-27 21:45:21.597756556 +0000 UTC m=+0.347976960,LastTimestamp:2026-07-27 21:45:21.597756556 +0000 UTC m=+0.347976960,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-80.ec2.internal,}" Jul 27 21:45:21.609997 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.609938 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-80.ec2.internal.18c643af17de8aa2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-80.ec2.internal,UID:ip-10-0-129-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-80.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-80.ec2.internal,},FirstTimestamp:2026-07-27 21:45:21.597778594 +0000 UTC m=+0.347998997,LastTimestamp:2026-07-27 21:45:21.597778594 +0000 UTC m=+0.347998997,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-80.ec2.internal,}" Jul 27 21:45:21.617840 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.617772 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-80.ec2.internal.18c643af17deada7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-80.ec2.internal,UID:ip-10-0-129-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-80.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-80.ec2.internal,},FirstTimestamp:2026-07-27 21:45:21.597787559 +0000 UTC m=+0.348007962,LastTimestamp:2026-07-27 21:45:21.597787559 +0000 UTC m=+0.348007962,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-80.ec2.internal,}" Jul 27 21:45:21.626864 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.626603 2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-80.ec2.internal.18c643af17de348c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-80.ec2.internal.18c643af17de348c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-80.ec2.internal,UID:ip-10-0-129-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-80.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-80.ec2.internal,},FirstTimestamp:2026-07-27 21:45:21.597756556 +0000 UTC m=+0.347976960,LastTimestamp:2026-07-27 21:45:21.599025322 +0000 UTC m=+0.349245725,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-80.ec2.internal,}" Jul 27 21:45:21.638624 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.638551 2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-80.ec2.internal.18c643af17de8aa2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-80.ec2.internal.18c643af17de8aa2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-80.ec2.internal,UID:ip-10-0-129-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-80.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-80.ec2.internal,},FirstTimestamp:2026-07-27 21:45:21.597778594 +0000 UTC m=+0.347998997,LastTimestamp:2026-07-27 21:45:21.59904579 +0000 UTC m=+0.349266192,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-80.ec2.internal,}" Jul 27 21:45:21.643091 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.643078 2561 manager.go:337] "Starting Device Plugin manager" Jul 27 21:45:21.643171 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.643117 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 21:45:21.643171 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.643131 2561 server.go:87] "Starting device plugin registration server" Jul 27 21:45:21.643393 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.643382 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 21:45:21.643456 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.643394 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 21:45:21.643518 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.643503 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 21:45:21.643608 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.643590 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 21:45:21.643608 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.643605 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 21:45:21.644094 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.644077 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 21:45:21.644160 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.644120 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-80.ec2.internal\" not found" Jul 27 21:45:21.646800 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.646733 2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-80.ec2.internal.18c643af17deada7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-80.ec2.internal.18c643af17deada7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-80.ec2.internal,UID:ip-10-0-129-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-80.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-80.ec2.internal,},FirstTimestamp:2026-07-27 21:45:21.597787559 +0000 UTC m=+0.348007962,LastTimestamp:2026-07-27 21:45:21.599055301 +0000 UTC m=+0.349275704,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-80.ec2.internal,}" Jul 27 21:45:21.657213 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.657160 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-80.ec2.internal.18c643af1aba0e2f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-80.ec2.internal,UID:ip-10-0-129-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-129-80.ec2.internal,},FirstTimestamp:2026-07-27 21:45:21.645719087 +0000 UTC m=+0.395939490,LastTimestamp:2026-07-27 21:45:21.645719087 +0000 UTC m=+0.395939490,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-80.ec2.internal,}" Jul 27 21:45:21.673864 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.673842 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q6fwj" Jul 27 21:45:21.680833 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.679217 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 21:45:21.680833 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.679256 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 21:45:21.680833 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.679277 2561 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 21:45:21.680833 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.679311 2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 21:45:21.681837 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.681818 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q6fwj" Jul 27 21:45:21.688928 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.688914 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:45:21.743993 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.743952 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:45:21.744941 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.744925 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:45:21.745018 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.744954 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:45:21.745018 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.744964 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:45:21.745018 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.744996 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-80.ec2.internal" Jul 27 21:45:21.754457 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.754442 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-80.ec2.internal" Jul 27 21:45:21.754531 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.754465 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-80.ec2.internal\": node \"ip-10-0-129-80.ec2.internal\" not found" Jul 27 21:45:21.772583 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.772510 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 27 21:45:21.779419 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.779393 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal"] Jul 27 21:45:21.779533 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.779464 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:45:21.781228 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.781210 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:45:21.781331 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.781236 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:45:21.781331 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.781246 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:45:21.782360 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.782341 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0483318e6ca30ead09c5152f256a0232-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"0483318e6ca30ead09c5152f256a0232\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 27 21:45:21.782449 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.782367 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0483318e6ca30ead09c5152f256a0232-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"0483318e6ca30ead09c5152f256a0232\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 27 21:45:21.782564 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.782541 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:45:21.782664 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.782649 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 27 21:45:21.782704 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.782678 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:45:21.784045 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.784028 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:45:21.784127 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.784034 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:45:21.784127 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.784074 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:45:21.784127 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.784086 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:45:21.784127 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.784054 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:45:21.784246 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.784129 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:45:21.785194 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.785181 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" Jul 27 21:45:21.785233 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.785204 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:45:21.786110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.786094 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:45:21.786176 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.786116 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:45:21.786176 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:21.786133 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:45:21.812117 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.812101 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-80.ec2.internal\" not found" node="ip-10-0-129-80.ec2.internal" Jul 27 21:45:21.816610 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:21.816595 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-80.ec2.internal\" not found" node="ip-10-0-129-80.ec2.internal" Jul 27 21:45:23.005927 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:23.005894 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 27 21:45:23.016107 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.016081 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0483318e6ca30ead09c5152f256a0232-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"0483318e6ca30ead09c5152f256a0232\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 27 21:45:23.016187 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.016111 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0483318e6ca30ead09c5152f256a0232-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"0483318e6ca30ead09c5152f256a0232\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 27 21:45:23.016187 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.016135 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f186ee96a5b10264dea34206a4faa43e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-80.ec2.internal\" (UID: \"f186ee96a5b10264dea34206a4faa43e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" Jul 27 21:45:23.016276 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.016190 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0483318e6ca30ead09c5152f256a0232-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"0483318e6ca30ead09c5152f256a0232\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 27 21:45:23.016276 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.016193 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0483318e6ca30ead09c5152f256a0232-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"0483318e6ca30ead09c5152f256a0232\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 27 21:45:23.106816 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:23.106781 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 27 21:45:23.117157 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.117132 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f186ee96a5b10264dea34206a4faa43e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-80.ec2.internal\" (UID: \"f186ee96a5b10264dea34206a4faa43e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" Jul 27 21:45:23.117226 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.117163 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f186ee96a5b10264dea34206a4faa43e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-80.ec2.internal\" (UID: \"f186ee96a5b10264dea34206a4faa43e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" Jul 27 21:45:23.207655 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:23.207593 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 27 21:45:23.248138 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.248104 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 27 21:45:23.252574 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.252250 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" Jul 27 21:45:23.308324 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:23.308293 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 27 21:45:23.408797 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:23.408767 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 27 21:45:23.509396 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:23.509322 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 27 21:45:23.561565 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.561539 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:45:23.609975 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:23.609942 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 27 21:45:23.661431 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.661390 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 21:45:23.661863 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.661572 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 21:45:23.661863 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.661599 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 21:45:23.705409 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.705379 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:45:23.710060 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:23.710039 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 27 21:45:23.782917 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.782839 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:45:23.784132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.784106 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:45:23.784256 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.784144 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:45:23.784256 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.784159 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:45:23.810391 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:23.810370 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 27 21:45:23.814140 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:23.814102 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf186ee96a5b10264dea34206a4faa43e.slice/crio-7c549ebed40ee3b9a3732af5d794c071da14d2a0c9cb70b2698928affec32e87 WatchSource:0}: Error finding container 7c549ebed40ee3b9a3732af5d794c071da14d2a0c9cb70b2698928affec32e87: Status 404 returned error can't find the container with id 7c549ebed40ee3b9a3732af5d794c071da14d2a0c9cb70b2698928affec32e87 Jul 27 21:45:23.814520 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:23.814497 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0483318e6ca30ead09c5152f256a0232.slice/crio-7061f9a1fa9769094b0e73a00dabf5f171949872b3ac5a8aed0ea843e69101c8 WatchSource:0}: Error finding container 7061f9a1fa9769094b0e73a00dabf5f171949872b3ac5a8aed0ea843e69101c8: Status 404 returned error can't find the container with id 7061f9a1fa9769094b0e73a00dabf5f171949872b3ac5a8aed0ea843e69101c8 Jul 27 21:45:23.818299 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.818256 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 21:40:22 +0000 UTC" deadline="2028-02-14 10:11:36.386844317 +0000 UTC" Jul 27 21:45:23.818353 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.818302 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13596h26m12.568547538s" Jul 27 21:45:23.818353 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.818320 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 21:45:23.850337 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.850309 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:45:23.913910 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.913887 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 27 21:45:23.924509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.924488 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 21:45:23.925410 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.925398 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" Jul 27 21:45:23.933418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:23.933399 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 21:45:24.694453 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.694424 2561 apiserver.go:52] "Watching apiserver" Jul 27 21:45:24.701235 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.700799 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 21:45:24.702353 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.702321 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-gqtgz","openshift-multus/network-metrics-daemon-p79rn","openshift-ovn-kubernetes/ovnkube-node-z4kt4","kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal","openshift-cluster-node-tuning-operator/tuned-h9t4q","openshift-image-registry/node-ca-ghrhz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal","openshift-network-diagnostics/network-check-target-w86pt","openshift-network-operator/iptables-alerter-m2lx2","kube-system/global-pull-secret-syncer-vlh9m","kube-system/konnectivity-agent-6hjdx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs","openshift-dns/node-resolver-8m8z2","openshift-multus/multus-additional-cni-plugins-9lnz2"] Jul 27 21:45:24.704995 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.704974 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.707479 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.707455 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 21:45:24.707578 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.707509 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 21:45:24.707934 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.707696 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 21:45:24.707934 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.707752 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7lhpd\"" Jul 27 21:45:24.708772 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.708748 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:24.708854 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:24.708822 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:24.710827 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.710811 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.713178 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.713154 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.714518 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.713942 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 21:45:24.714518 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.714056 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 21:45:24.714518 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.714120 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 21:45:24.714518 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.714227 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 21:45:24.714518 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.714315 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n79rg\"" Jul 27 21:45:24.714518 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.714457 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 21:45:24.714518 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.714462 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 21:45:24.714833 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.714578 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 21:45:24.715498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.715480 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.715594 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.715552 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 21:45:24.716154 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.715798 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 21:45:24.716154 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.715926 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 21:45:24.716154 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.715942 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lt4gx\"" Jul 27 21:45:24.716154 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.716009 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 21:45:24.716154 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.716009 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 21:45:24.717360 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.717336 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:45:24.717529 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.717498 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4p4hl\"" Jul 27 21:45:24.717630 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.717613 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 21:45:24.717782 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.717767 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ghrhz" Jul 27 21:45:24.719458 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.719438 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 21:45:24.719659 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.719641 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 21:45:24.719729 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.719652 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fccn4\"" Jul 27 21:45:24.719936 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.719917 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 21:45:24.720088 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.720071 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.722172 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.722154 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 21:45:24.722344 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.722326 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qjf94\"" Jul 27 21:45:24.723709 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.723685 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:24.723841 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:24.723820 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:24.725333 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725312 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.725431 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725346 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a847c2b0-e057-481f-8cf2-14082d3fffd0-env-overrides\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.725431 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725372 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-run\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.725431 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725408 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-sys\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.725431 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725430 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.725624 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725444 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-run-systemd\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.725624 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725464 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-run-openvswitch\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.725624 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725515 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-cni-netd\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.725624 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725546 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a847c2b0-e057-481f-8cf2-14082d3fffd0-ovnkube-config\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.725624 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725591 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a847c2b0-e057-481f-8cf2-14082d3fffd0-ovnkube-script-lib\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.725624 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725622 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-cni-bin\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.725927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725665 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-os-release\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.725927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725691 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-sysctl-d\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.725927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725713 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-systemd\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.725927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725735 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-var-lib-kubelet\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.725927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725757 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpj6c\" (UniqueName: \"kubernetes.io/projected/97dc9818-711d-4003-8f27-7bde8cf8176b-kube-api-access-vpj6c\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.725927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725780 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-device-dir\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.725927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725834 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.725927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725860 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-modprobe-d\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.725927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725909 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-sysctl-conf\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725953 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-lib-modules\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.725986 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65d8s\" (UniqueName: \"kubernetes.io/projected/df4b987a-623d-4079-bbb9-b38e0fa986f5-kube-api-access-65d8s\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726031 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lfn\" (UniqueName: \"kubernetes.io/projected/bf30d435-7854-4c24-b72a-c6992b0776a7-kube-api-access-k9lfn\") pod \"network-metrics-daemon-p79rn\" (UID: \"bf30d435-7854-4c24-b72a-c6992b0776a7\") " pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726064 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-etc-openvswitch\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726088 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-log-socket\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726119 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a847c2b0-e057-481f-8cf2-14082d3fffd0-ovn-node-metrics-cert\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726140 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m2lx2" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726145 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zqpv\" (UniqueName: \"kubernetes.io/projected/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-kube-api-access-7zqpv\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726190 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97dc9818-711d-4003-8f27-7bde8cf8176b-tmp\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726231 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-run-ovn-kubernetes\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726279 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7wp4\" (UniqueName: \"kubernetes.io/projected/a847c2b0-e057-481f-8cf2-14082d3fffd0-kube-api-access-v7wp4\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726318 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726343 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.726418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726372 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-sysconfig\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726433 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs\") pod \"network-metrics-daemon-p79rn\" (UID: \"bf30d435-7854-4c24-b72a-c6992b0776a7\") " pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726477 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-slash\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726499 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-var-lib-openvswitch\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726530 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-cnibin\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726570 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726595 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-host\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726618 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-tuned\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726666 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-registration-dir\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726691 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-sys-fs\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726700 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726718 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-kubelet\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726754 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-systemd-units\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726781 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-run-ovn\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726804 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-system-cni-dir\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726832 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.727095 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726859 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-kubernetes\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.727841 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726882 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-socket-dir\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.727841 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726907 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-run-netns\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.727841 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.726931 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-node-log\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.728068 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.728047 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:45:24.728307 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.728289 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g4ffb\"" Jul 27 21:45:24.728505 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.728490 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 21:45:24.728721 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.728706 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 21:45:24.728878 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.728864 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8m8z2" Jul 27 21:45:24.730792 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.730722 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 21:45:24.730898 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.730797 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jf6d4\"" Jul 27 21:45:24.731186 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.731171 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 21:45:24.731649 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.731583 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6hjdx" Jul 27 21:45:24.733462 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.733444 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 21:45:24.733555 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.733514 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 21:45:24.733611 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.733444 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6bcsm\"" Jul 27 21:45:24.733740 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.733722 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:24.733812 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:24.733791 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:24.751796 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.751764 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6ts7z" Jul 27 21:45:24.757929 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.757910 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6ts7z" Jul 27 21:45:24.815279 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.815244 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 21:45:24.817036 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.816987 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" event={"ID":"f186ee96a5b10264dea34206a4faa43e","Type":"ContainerStarted","Data":"7c549ebed40ee3b9a3732af5d794c071da14d2a0c9cb70b2698928affec32e87"} Jul 27 21:45:24.818163 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.818123 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" event={"ID":"0483318e6ca30ead09c5152f256a0232","Type":"ContainerStarted","Data":"7061f9a1fa9769094b0e73a00dabf5f171949872b3ac5a8aed0ea843e69101c8"} Jul 27 21:45:24.827762 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.827737 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-systemd\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.827875 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.827778 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g872m\" (UniqueName: \"kubernetes.io/projected/e6c2bede-f097-4c04-8a7b-1c1e83d9d284-kube-api-access-g872m\") pod \"iptables-alerter-m2lx2\" (UID: \"e6c2bede-f097-4c04-8a7b-1c1e83d9d284\") " pod="openshift-network-operator/iptables-alerter-m2lx2" Jul 27 21:45:24.827875 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.827807 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f949673-96be-4e7f-9cf8-5f69124a6309-hosts-file\") pod \"node-resolver-8m8z2\" (UID: \"7f949673-96be-4e7f-9cf8-5f69124a6309\") " pod="openshift-dns/node-resolver-8m8z2" Jul 27 21:45:24.827875 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.827833 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.827875 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.827839 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-systemd\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.827875 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.827855 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-run-k8s-cni-cncf-io\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.828110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.827877 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-var-lib-cni-bin\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.828110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.827900 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f919d24f-2d1e-4b16-b946-342af47fe64c-multus-daemon-config\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.828110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.827919 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-run-multus-certs\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.828110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.827941 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvgq\" (UniqueName: \"kubernetes.io/projected/7f949673-96be-4e7f-9cf8-5f69124a6309-kube-api-access-tjvgq\") pod \"node-resolver-8m8z2\" (UID: \"7f949673-96be-4e7f-9cf8-5f69124a6309\") " pod="openshift-dns/node-resolver-8m8z2" Jul 27 21:45:24.828110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.827964 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65d8s\" (UniqueName: \"kubernetes.io/projected/df4b987a-623d-4079-bbb9-b38e0fa986f5-kube-api-access-65d8s\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.828110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.827981 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.828110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.827989 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lfn\" (UniqueName: \"kubernetes.io/projected/bf30d435-7854-4c24-b72a-c6992b0776a7-kube-api-access-k9lfn\") pod \"network-metrics-daemon-p79rn\" (UID: \"bf30d435-7854-4c24-b72a-c6992b0776a7\") " pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:24.828110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828014 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a847c2b0-e057-481f-8cf2-14082d3fffd0-ovn-node-metrics-cert\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.828110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828039 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.828110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828062 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-sysconfig\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.828110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828086 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f949673-96be-4e7f-9cf8-5f69124a6309-tmp-dir\") pod \"node-resolver-8m8z2\" (UID: \"7f949673-96be-4e7f-9cf8-5f69124a6309\") " pod="openshift-dns/node-resolver-8m8z2" Jul 27 21:45:24.828110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828108 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-slash\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828131 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828162 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-host\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828183 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-run-netns\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828207 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-registration-dir\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828230 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-kubelet\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828253 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-systemd-units\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828291 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-run-ovn\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828338 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-slash\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828351 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-kubelet\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828370 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-system-cni-dir\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828392 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828417 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-kubernetes\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828415 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-host\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828440 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6aaee722-f9e3-4563-addf-2a4fe9f5f0a0-konnectivity-ca\") pod \"konnectivity-agent-6hjdx\" (UID: \"6aaee722-f9e3-4563-addf-2a4fe9f5f0a0\") " pod="kube-system/konnectivity-agent-6hjdx" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828456 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-socket-dir\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828471 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-run-netns\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.828665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828471 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-system-cni-dir\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828483 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-kubernetes\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828498 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-node-log\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828514 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828513 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828530 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a847c2b0-e057-481f-8cf2-14082d3fffd0-env-overrides\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828545 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8aa2c150-1b16-43c8-81a7-a290299036d3-host\") pod \"node-ca-ghrhz\" (UID: \"8aa2c150-1b16-43c8-81a7-a290299036d3\") " pod="openshift-image-registry/node-ca-ghrhz" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828545 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-node-log\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828563 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828602 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-registration-dir\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828617 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-multus-socket-dir-parent\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828648 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-socket-dir\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828648 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-run-systemd\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828692 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-run-systemd\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828715 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-run-openvswitch\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828743 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-run-netns\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828776 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-cni-netd\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.829461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828803 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a847c2b0-e057-481f-8cf2-14082d3fffd0-ovnkube-config\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828829 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a847c2b0-e057-481f-8cf2-14082d3fffd0-ovnkube-script-lib\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828830 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-run-openvswitch\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828876 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrp7\" (UniqueName: \"kubernetes.io/projected/f919d24f-2d1e-4b16-b946-342af47fe64c-kube-api-access-mjrp7\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828887 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-cni-netd\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828895 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828913 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-run-ovn\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828925 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6c2bede-f097-4c04-8a7b-1c1e83d9d284-host-slash\") pod \"iptables-alerter-m2lx2\" (UID: \"e6c2bede-f097-4c04-8a7b-1c1e83d9d284\") " pod="openshift-network-operator/iptables-alerter-m2lx2" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828953 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-systemd-units\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.828972 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-os-release\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829017 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-var-lib-kubelet\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829032 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a847c2b0-e057-481f-8cf2-14082d3fffd0-env-overrides\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829043 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpj6c\" (UniqueName: \"kubernetes.io/projected/97dc9818-711d-4003-8f27-7bde8cf8176b-kube-api-access-vpj6c\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829066 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-device-dir\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829093 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-modprobe-d\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829134 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-sysconfig\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829135 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.830281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829138 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829155 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-sysctl-conf\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829196 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-os-release\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829204 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-var-lib-kubelet\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829187 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-lib-modules\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829288 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzgws\" (UniqueName: \"kubernetes.io/projected/8aa2c150-1b16-43c8-81a7-a290299036d3-kube-api-access-jzgws\") pod \"node-ca-ghrhz\" (UID: \"8aa2c150-1b16-43c8-81a7-a290299036d3\") " pod="openshift-image-registry/node-ca-ghrhz" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829318 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-cnibin\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829335 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-lib-modules\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829370 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-device-dir\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829419 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a847c2b0-e057-481f-8cf2-14082d3fffd0-ovnkube-config\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829483 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-os-release\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829517 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a847c2b0-e057-481f-8cf2-14082d3fffd0-ovnkube-script-lib\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829519 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ef19975-ecaa-49ba-9cd9-e16737e46c47-kubelet-config\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829552 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-etc-openvswitch\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829553 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-sysctl-conf\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829555 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-modprobe-d\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829594 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-log-socket\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.831132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829596 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-etc-openvswitch\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829620 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zqpv\" (UniqueName: \"kubernetes.io/projected/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-kube-api-access-7zqpv\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829651 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-log-socket\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829701 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97dc9818-711d-4003-8f27-7bde8cf8176b-tmp\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829731 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-hostroot\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829810 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829841 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-run-ovn-kubernetes\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829868 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7wp4\" (UniqueName: \"kubernetes.io/projected/a847c2b0-e057-481f-8cf2-14082d3fffd0-kube-api-access-v7wp4\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829894 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829943 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-var-lib-cni-multus\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829968 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-multus-conf-dir\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.829991 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ef19975-ecaa-49ba-9cd9-e16737e46c47-dbus\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830016 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs\") pod \"network-metrics-daemon-p79rn\" (UID: \"bf30d435-7854-4c24-b72a-c6992b0776a7\") " pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830042 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-var-lib-openvswitch\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830070 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-cnibin\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830094 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-tuned\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830117 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f919d24f-2d1e-4b16-b946-342af47fe64c-cni-binary-copy\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.831990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830207 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-var-lib-kubelet\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830241 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-sys-fs\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830277 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-cnibin\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830279 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-run\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830342 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-var-lib-openvswitch\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830399 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-run\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:24.830429 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830431 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-sys\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830427 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-sys-fs\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830461 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-run-ovn-kubernetes\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830518 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-sys\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:24.830609 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs podName:bf30d435-7854-4c24-b72a-c6992b0776a7 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:25.330570278 +0000 UTC m=+2.948007579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs") pod "network-metrics-daemon-p79rn" (UID: "bf30d435-7854-4c24-b72a-c6992b0776a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830630 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-multus-cni-dir\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830658 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-etc-kubernetes\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830682 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8aa2c150-1b16-43c8-81a7-a290299036d3-serviceca\") pod \"node-ca-ghrhz\" (UID: \"8aa2c150-1b16-43c8-81a7-a290299036d3\") " pod="openshift-image-registry/node-ca-ghrhz" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830705 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6aaee722-f9e3-4563-addf-2a4fe9f5f0a0-agent-certs\") pod \"konnectivity-agent-6hjdx\" (UID: \"6aaee722-f9e3-4563-addf-2a4fe9f5f0a0\") " pod="kube-system/konnectivity-agent-6hjdx" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.830903 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.834042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.831094 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-692z2\" (UniqueName: \"kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2\") pod \"network-check-target-w86pt\" (UID: \"5bdac9f5-97c0-4a7f-9ec7-dd07f0343253\") " pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:24.834989 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.831130 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e6c2bede-f097-4c04-8a7b-1c1e83d9d284-iptables-alerter-script\") pod \"iptables-alerter-m2lx2\" (UID: \"e6c2bede-f097-4c04-8a7b-1c1e83d9d284\") " pod="openshift-network-operator/iptables-alerter-m2lx2" Jul 27 21:45:24.834989 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.831181 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.834989 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.831234 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-system-cni-dir\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.834989 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.831322 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df4b987a-623d-4079-bbb9-b38e0fa986f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.834989 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.831379 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-cni-bin\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.834989 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.831422 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-sysctl-d\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.834989 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.831435 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a847c2b0-e057-481f-8cf2-14082d3fffd0-host-cni-bin\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.834989 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.831562 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-sysctl-d\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.834989 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.833170 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97dc9818-711d-4003-8f27-7bde8cf8176b-tmp\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.834989 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.833414 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a847c2b0-e057-481f-8cf2-14082d3fffd0-ovn-node-metrics-cert\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.834989 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.833921 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97dc9818-711d-4003-8f27-7bde8cf8176b-etc-tuned\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.835771 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.835707 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65d8s\" (UniqueName: \"kubernetes.io/projected/df4b987a-623d-4079-bbb9-b38e0fa986f5-kube-api-access-65d8s\") pod \"aws-ebs-csi-driver-node-f42cs\" (UID: \"df4b987a-623d-4079-bbb9-b38e0fa986f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:24.836708 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.836688 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lfn\" (UniqueName: \"kubernetes.io/projected/bf30d435-7854-4c24-b72a-c6992b0776a7-kube-api-access-k9lfn\") pod \"network-metrics-daemon-p79rn\" (UID: \"bf30d435-7854-4c24-b72a-c6992b0776a7\") " pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:24.838182 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.838098 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpj6c\" (UniqueName: \"kubernetes.io/projected/97dc9818-711d-4003-8f27-7bde8cf8176b-kube-api-access-vpj6c\") pod \"tuned-h9t4q\" (UID: \"97dc9818-711d-4003-8f27-7bde8cf8176b\") " pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:24.841501 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.841434 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zqpv\" (UniqueName: \"kubernetes.io/projected/4eb68d22-0e6c-4291-9657-fa19b31b6cd6-kube-api-access-7zqpv\") pod \"multus-additional-cni-plugins-9lnz2\" (UID: \"4eb68d22-0e6c-4291-9657-fa19b31b6cd6\") " pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:24.841501 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.841445 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7wp4\" (UniqueName: \"kubernetes.io/projected/a847c2b0-e057-481f-8cf2-14082d3fffd0-kube-api-access-v7wp4\") pod \"ovnkube-node-z4kt4\" (UID: \"a847c2b0-e057-481f-8cf2-14082d3fffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:24.932684 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.932656 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-var-lib-kubelet\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.932879 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.932696 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-multus-cni-dir\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.932879 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.932715 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-etc-kubernetes\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.932879 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.932737 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8aa2c150-1b16-43c8-81a7-a290299036d3-serviceca\") pod \"node-ca-ghrhz\" (UID: \"8aa2c150-1b16-43c8-81a7-a290299036d3\") " pod="openshift-image-registry/node-ca-ghrhz" Jul 27 21:45:24.932879 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.932769 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-var-lib-kubelet\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.932879 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.932798 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-etc-kubernetes\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.932879 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.932849 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6aaee722-f9e3-4563-addf-2a4fe9f5f0a0-agent-certs\") pod \"konnectivity-agent-6hjdx\" (UID: \"6aaee722-f9e3-4563-addf-2a4fe9f5f0a0\") " pod="kube-system/konnectivity-agent-6hjdx" Jul 27 21:45:24.933170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.932882 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-multus-cni-dir\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.932898 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692z2\" (UniqueName: \"kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2\") pod \"network-check-target-w86pt\" (UID: \"5bdac9f5-97c0-4a7f-9ec7-dd07f0343253\") " pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:24.933170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.932926 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e6c2bede-f097-4c04-8a7b-1c1e83d9d284-iptables-alerter-script\") pod \"iptables-alerter-m2lx2\" (UID: \"e6c2bede-f097-4c04-8a7b-1c1e83d9d284\") " pod="openshift-network-operator/iptables-alerter-m2lx2" Jul 27 21:45:24.933170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.932952 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-system-cni-dir\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.932981 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g872m\" (UniqueName: \"kubernetes.io/projected/e6c2bede-f097-4c04-8a7b-1c1e83d9d284-kube-api-access-g872m\") pod \"iptables-alerter-m2lx2\" (UID: \"e6c2bede-f097-4c04-8a7b-1c1e83d9d284\") " pod="openshift-network-operator/iptables-alerter-m2lx2" Jul 27 21:45:24.933170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933004 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f949673-96be-4e7f-9cf8-5f69124a6309-hosts-file\") pod \"node-resolver-8m8z2\" (UID: \"7f949673-96be-4e7f-9cf8-5f69124a6309\") " pod="openshift-dns/node-resolver-8m8z2" Jul 27 21:45:24.933170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933031 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-run-k8s-cni-cncf-io\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933055 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-var-lib-cni-bin\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933078 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f919d24f-2d1e-4b16-b946-342af47fe64c-multus-daemon-config\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933103 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-run-multus-certs\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933126 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvgq\" (UniqueName: \"kubernetes.io/projected/7f949673-96be-4e7f-9cf8-5f69124a6309-kube-api-access-tjvgq\") pod \"node-resolver-8m8z2\" (UID: \"7f949673-96be-4e7f-9cf8-5f69124a6309\") " pod="openshift-dns/node-resolver-8m8z2" Jul 27 21:45:24.933170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933155 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f949673-96be-4e7f-9cf8-5f69124a6309-tmp-dir\") pod \"node-resolver-8m8z2\" (UID: \"7f949673-96be-4e7f-9cf8-5f69124a6309\") " pod="openshift-dns/node-resolver-8m8z2" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933174 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8aa2c150-1b16-43c8-81a7-a290299036d3-serviceca\") pod \"node-ca-ghrhz\" (UID: \"8aa2c150-1b16-43c8-81a7-a290299036d3\") " pod="openshift-image-registry/node-ca-ghrhz" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933182 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-run-netns\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933225 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6aaee722-f9e3-4563-addf-2a4fe9f5f0a0-konnectivity-ca\") pod \"konnectivity-agent-6hjdx\" (UID: \"6aaee722-f9e3-4563-addf-2a4fe9f5f0a0\") " pod="kube-system/konnectivity-agent-6hjdx" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933254 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8aa2c150-1b16-43c8-81a7-a290299036d3-host\") pod \"node-ca-ghrhz\" (UID: \"8aa2c150-1b16-43c8-81a7-a290299036d3\") " pod="openshift-image-registry/node-ca-ghrhz" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933285 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-system-cni-dir\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933292 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-multus-socket-dir-parent\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933339 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrp7\" (UniqueName: \"kubernetes.io/projected/f919d24f-2d1e-4b16-b946-342af47fe64c-kube-api-access-mjrp7\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933350 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-multus-socket-dir-parent\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933521 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6c2bede-f097-4c04-8a7b-1c1e83d9d284-host-slash\") pod \"iptables-alerter-m2lx2\" (UID: \"e6c2bede-f097-4c04-8a7b-1c1e83d9d284\") " pod="openshift-network-operator/iptables-alerter-m2lx2" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933567 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzgws\" (UniqueName: \"kubernetes.io/projected/8aa2c150-1b16-43c8-81a7-a290299036d3-kube-api-access-jzgws\") pod \"node-ca-ghrhz\" (UID: \"8aa2c150-1b16-43c8-81a7-a290299036d3\") " pod="openshift-image-registry/node-ca-ghrhz" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933593 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-cnibin\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933616 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-os-release\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933623 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-run-netns\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933639 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f949673-96be-4e7f-9cf8-5f69124a6309-hosts-file\") pod \"node-resolver-8m8z2\" (UID: \"7f949673-96be-4e7f-9cf8-5f69124a6309\") " pod="openshift-dns/node-resolver-8m8z2" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933660 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6c2bede-f097-4c04-8a7b-1c1e83d9d284-host-slash\") pod \"iptables-alerter-m2lx2\" (UID: \"e6c2bede-f097-4c04-8a7b-1c1e83d9d284\") " pod="openshift-network-operator/iptables-alerter-m2lx2" Jul 27 21:45:24.933745 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933684 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-run-k8s-cni-cncf-io\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933835 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f919d24f-2d1e-4b16-b946-342af47fe64c-multus-daemon-config\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933852 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-cnibin\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933596 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f949673-96be-4e7f-9cf8-5f69124a6309-tmp-dir\") pod \"node-resolver-8m8z2\" (UID: \"7f949673-96be-4e7f-9cf8-5f69124a6309\") " pod="openshift-dns/node-resolver-8m8z2" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933881 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ef19975-ecaa-49ba-9cd9-e16737e46c47-kubelet-config\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933902 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8aa2c150-1b16-43c8-81a7-a290299036d3-host\") pod \"node-ca-ghrhz\" (UID: \"8aa2c150-1b16-43c8-81a7-a290299036d3\") " pod="openshift-image-registry/node-ca-ghrhz" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933904 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-hostroot\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933926 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-hostroot\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933933 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-os-release\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933226 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-var-lib-cni-bin\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933948 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933958 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ef19975-ecaa-49ba-9cd9-e16737e46c47-kubelet-config\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933979 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-var-lib-cni-multus\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.933982 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-run-multus-certs\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.934006 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-multus-conf-dir\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.934031 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-host-var-lib-cni-multus\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.934038 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ef19975-ecaa-49ba-9cd9-e16737e46c47-dbus\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:24.934057 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.934068 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f919d24f-2d1e-4b16-b946-342af47fe64c-multus-conf-dir\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.934520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.934080 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f919d24f-2d1e-4b16-b946-342af47fe64c-cni-binary-copy\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.935202 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.934128 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ef19975-ecaa-49ba-9cd9-e16737e46c47-dbus\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:24.935202 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:24.934223 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret podName:1ef19975-ecaa-49ba-9cd9-e16737e46c47 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:25.434197553 +0000 UTC m=+3.051634857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret") pod "global-pull-secret-syncer-vlh9m" (UID: "1ef19975-ecaa-49ba-9cd9-e16737e46c47") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:24.935202 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.934544 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e6c2bede-f097-4c04-8a7b-1c1e83d9d284-iptables-alerter-script\") pod \"iptables-alerter-m2lx2\" (UID: \"e6c2bede-f097-4c04-8a7b-1c1e83d9d284\") " pod="openshift-network-operator/iptables-alerter-m2lx2" Jul 27 21:45:24.935202 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.934633 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f919d24f-2d1e-4b16-b946-342af47fe64c-cni-binary-copy\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.935202 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.935078 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6aaee722-f9e3-4563-addf-2a4fe9f5f0a0-konnectivity-ca\") pod \"konnectivity-agent-6hjdx\" (UID: \"6aaee722-f9e3-4563-addf-2a4fe9f5f0a0\") " pod="kube-system/konnectivity-agent-6hjdx" Jul 27 21:45:24.935761 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.935736 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6aaee722-f9e3-4563-addf-2a4fe9f5f0a0-agent-certs\") pod \"konnectivity-agent-6hjdx\" (UID: \"6aaee722-f9e3-4563-addf-2a4fe9f5f0a0\") " pod="kube-system/konnectivity-agent-6hjdx" Jul 27 21:45:24.941778 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:24.941553 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:45:24.941778 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:24.941576 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:45:24.941778 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:24.941588 2561 projected.go:196] Error preparing data for projected volume kube-api-access-692z2 for pod openshift-network-diagnostics/network-check-target-w86pt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:24.941778 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:24.941646 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2 podName:5bdac9f5-97c0-4a7f-9ec7-dd07f0343253 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:25.441633897 +0000 UTC m=+3.059071194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-692z2" (UniqueName: "kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2") pod "network-check-target-w86pt" (UID: "5bdac9f5-97c0-4a7f-9ec7-dd07f0343253") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:24.943977 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.943956 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvgq\" (UniqueName: \"kubernetes.io/projected/7f949673-96be-4e7f-9cf8-5f69124a6309-kube-api-access-tjvgq\") pod \"node-resolver-8m8z2\" (UID: \"7f949673-96be-4e7f-9cf8-5f69124a6309\") " pod="openshift-dns/node-resolver-8m8z2" Jul 27 21:45:24.944077 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.944063 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrp7\" (UniqueName: \"kubernetes.io/projected/f919d24f-2d1e-4b16-b946-342af47fe64c-kube-api-access-mjrp7\") pod \"multus-gqtgz\" (UID: \"f919d24f-2d1e-4b16-b946-342af47fe64c\") " pod="openshift-multus/multus-gqtgz" Jul 27 21:45:24.944328 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.944300 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzgws\" (UniqueName: \"kubernetes.io/projected/8aa2c150-1b16-43c8-81a7-a290299036d3-kube-api-access-jzgws\") pod \"node-ca-ghrhz\" (UID: \"8aa2c150-1b16-43c8-81a7-a290299036d3\") " pod="openshift-image-registry/node-ca-ghrhz" Jul 27 21:45:24.944409 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:24.944339 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g872m\" (UniqueName: \"kubernetes.io/projected/e6c2bede-f097-4c04-8a7b-1c1e83d9d284-kube-api-access-g872m\") pod \"iptables-alerter-m2lx2\" (UID: \"e6c2bede-f097-4c04-8a7b-1c1e83d9d284\") " pod="openshift-network-operator/iptables-alerter-m2lx2" Jul 27 21:45:25.018832 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.018753 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" Jul 27 21:45:25.027333 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.027303 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:25.037254 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.037232 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9lnz2" Jul 27 21:45:25.047016 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.046991 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" Jul 27 21:45:25.054802 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.054778 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ghrhz" Jul 27 21:45:25.062665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.062648 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gqtgz" Jul 27 21:45:25.069042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.069023 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m2lx2" Jul 27 21:45:25.078371 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.078346 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8m8z2" Jul 27 21:45:25.083912 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.083889 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6hjdx" Jul 27 21:45:25.138833 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.138805 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:45:25.336988 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.336914 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs\") pod \"network-metrics-daemon-p79rn\" (UID: \"bf30d435-7854-4c24-b72a-c6992b0776a7\") " pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:25.337154 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:25.337065 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:25.337154 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:25.337133 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs podName:bf30d435-7854-4c24-b72a-c6992b0776a7 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:26.3371165 +0000 UTC m=+3.954553804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs") pod "network-metrics-daemon-p79rn" (UID: "bf30d435-7854-4c24-b72a-c6992b0776a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:25.375170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.375133 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:45:25.437334 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.437304 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:25.437487 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:25.437470 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:25.437541 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:25.437530 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret podName:1ef19975-ecaa-49ba-9cd9-e16737e46c47 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:26.437514052 +0000 UTC m=+4.054951348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret") pod "global-pull-secret-syncer-vlh9m" (UID: "1ef19975-ecaa-49ba-9cd9-e16737e46c47") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:25.538334 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.538294 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692z2\" (UniqueName: \"kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2\") pod \"network-check-target-w86pt\" (UID: \"5bdac9f5-97c0-4a7f-9ec7-dd07f0343253\") " pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:25.538515 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:25.538463 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:45:25.538515 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:25.538482 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:45:25.538515 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:25.538491 2561 projected.go:196] Error preparing data for projected volume kube-api-access-692z2 for pod openshift-network-diagnostics/network-check-target-w86pt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:25.538644 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:25.538564 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2 podName:5bdac9f5-97c0-4a7f-9ec7-dd07f0343253 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:26.538544646 +0000 UTC m=+4.155981951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-692z2" (UniqueName: "kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2") pod "network-check-target-w86pt" (UID: "5bdac9f5-97c0-4a7f-9ec7-dd07f0343253") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:25.731818 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:25.731784 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa2c150_1b16_43c8_81a7_a290299036d3.slice/crio-cdf54b8449c41f42fed41a6da355c0508403c3bcf64f449c4db46ce36a251486 WatchSource:0}: Error finding container cdf54b8449c41f42fed41a6da355c0508403c3bcf64f449c4db46ce36a251486: Status 404 returned error can't find the container with id cdf54b8449c41f42fed41a6da355c0508403c3bcf64f449c4db46ce36a251486 Jul 27 21:45:25.753364 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:25.753327 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97dc9818_711d_4003_8f27_7bde8cf8176b.slice/crio-f4631a5ff8725c975ba79defee7f2fad4b588830c6661a48dbcd859a2bc7954b WatchSource:0}: Error finding container f4631a5ff8725c975ba79defee7f2fad4b588830c6661a48dbcd859a2bc7954b: Status 404 returned error can't find the container with id f4631a5ff8725c975ba79defee7f2fad4b588830c6661a48dbcd859a2bc7954b Jul 27 21:45:25.754048 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:25.754022 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf919d24f_2d1e_4b16_b946_342af47fe64c.slice/crio-fd1de29e4f985032ef1de830d5444a95e0fe2f384083da0734c9bf718d5f55a0 WatchSource:0}: Error finding container fd1de29e4f985032ef1de830d5444a95e0fe2f384083da0734c9bf718d5f55a0: Status 404 returned error can't find the container with id fd1de29e4f985032ef1de830d5444a95e0fe2f384083da0734c9bf718d5f55a0 Jul 27 21:45:25.759068 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:25.759043 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb68d22_0e6c_4291_9657_fa19b31b6cd6.slice/crio-bf15f5883580094ee8b9682df95391caf0e334d4360dcfe53cc614fe81c38880 WatchSource:0}: Error finding container bf15f5883580094ee8b9682df95391caf0e334d4360dcfe53cc614fe81c38880: Status 404 returned error can't find the container with id bf15f5883580094ee8b9682df95391caf0e334d4360dcfe53cc614fe81c38880 Jul 27 21:45:25.759670 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:25.759637 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f949673_96be_4e7f_9cf8_5f69124a6309.slice/crio-8c78b45691d16c00dee1b2e8da3444f2bcf8e8fa82c876b361d3f42e4d4a4092 WatchSource:0}: Error finding container 8c78b45691d16c00dee1b2e8da3444f2bcf8e8fa82c876b361d3f42e4d4a4092: Status 404 returned error can't find the container with id 8c78b45691d16c00dee1b2e8da3444f2bcf8e8fa82c876b361d3f42e4d4a4092 Jul 27 21:45:25.760711 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.760681 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 21:40:24 +0000 UTC" deadline="2028-03-03 16:08:11.112721386 +0000 UTC" Jul 27 21:45:25.760769 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.760713 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14034h22m45.352012833s" Jul 27 21:45:25.760850 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:25.760832 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda847c2b0_e057_481f_8cf2_14082d3fffd0.slice/crio-aaaa45078d6439951ca2a5f53127ccefa6b3091b0791c0f0aac9aeb7960cf13c WatchSource:0}: Error finding container aaaa45078d6439951ca2a5f53127ccefa6b3091b0791c0f0aac9aeb7960cf13c: Status 404 returned error can't find the container with id aaaa45078d6439951ca2a5f53127ccefa6b3091b0791c0f0aac9aeb7960cf13c Jul 27 21:45:25.761720 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:25.761698 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aaee722_f9e3_4563_addf_2a4fe9f5f0a0.slice/crio-f740c043c90cea868671dba7a2fc585679fa25d02872a94ec0669c4f143a145b WatchSource:0}: Error finding container f740c043c90cea868671dba7a2fc585679fa25d02872a94ec0669c4f143a145b: Status 404 returned error can't find the container with id f740c043c90cea868671dba7a2fc585679fa25d02872a94ec0669c4f143a145b Jul 27 21:45:25.763600 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:25.763546 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4b987a_623d_4079_bbb9_b38e0fa986f5.slice/crio-264b6f6c40cc2d5c12b840a70c8764c386664ab015b1ffd0c8107f137cfa7560 WatchSource:0}: Error finding container 264b6f6c40cc2d5c12b840a70c8764c386664ab015b1ffd0c8107f137cfa7560: Status 404 returned error can't find the container with id 264b6f6c40cc2d5c12b840a70c8764c386664ab015b1ffd0c8107f137cfa7560 Jul 27 21:45:25.764044 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:25.764021 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c2bede_f097_4c04_8a7b_1c1e83d9d284.slice/crio-60a9c147dc5ca1a8ee58fada5d5bb2fee6dd65cfac4afa268465e9821f200a8f WatchSource:0}: Error finding container 60a9c147dc5ca1a8ee58fada5d5bb2fee6dd65cfac4afa268465e9821f200a8f: Status 404 returned error can't find the container with id 60a9c147dc5ca1a8ee58fada5d5bb2fee6dd65cfac4afa268465e9821f200a8f Jul 27 21:45:25.820665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.820630 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8m8z2" event={"ID":"7f949673-96be-4e7f-9cf8-5f69124a6309","Type":"ContainerStarted","Data":"8c78b45691d16c00dee1b2e8da3444f2bcf8e8fa82c876b361d3f42e4d4a4092"} Jul 27 21:45:25.821935 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.821880 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gqtgz" event={"ID":"f919d24f-2d1e-4b16-b946-342af47fe64c","Type":"ContainerStarted","Data":"fd1de29e4f985032ef1de830d5444a95e0fe2f384083da0734c9bf718d5f55a0"} Jul 27 21:45:25.823020 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.822995 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" event={"ID":"97dc9818-711d-4003-8f27-7bde8cf8176b","Type":"ContainerStarted","Data":"f4631a5ff8725c975ba79defee7f2fad4b588830c6661a48dbcd859a2bc7954b"} Jul 27 21:45:25.824934 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.824468 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m2lx2" event={"ID":"e6c2bede-f097-4c04-8a7b-1c1e83d9d284","Type":"ContainerStarted","Data":"60a9c147dc5ca1a8ee58fada5d5bb2fee6dd65cfac4afa268465e9821f200a8f"} Jul 27 21:45:25.826647 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.826329 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" event={"ID":"df4b987a-623d-4079-bbb9-b38e0fa986f5","Type":"ContainerStarted","Data":"264b6f6c40cc2d5c12b840a70c8764c386664ab015b1ffd0c8107f137cfa7560"} Jul 27 21:45:25.828174 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.828155 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lnz2" event={"ID":"4eb68d22-0e6c-4291-9657-fa19b31b6cd6","Type":"ContainerStarted","Data":"bf15f5883580094ee8b9682df95391caf0e334d4360dcfe53cc614fe81c38880"} Jul 27 21:45:25.829083 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.829055 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ghrhz" event={"ID":"8aa2c150-1b16-43c8-81a7-a290299036d3","Type":"ContainerStarted","Data":"cdf54b8449c41f42fed41a6da355c0508403c3bcf64f449c4db46ce36a251486"} Jul 27 21:45:25.830171 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.830146 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6hjdx" event={"ID":"6aaee722-f9e3-4563-addf-2a4fe9f5f0a0","Type":"ContainerStarted","Data":"f740c043c90cea868671dba7a2fc585679fa25d02872a94ec0669c4f143a145b"} Jul 27 21:45:25.831239 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:25.831220 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" event={"ID":"a847c2b0-e057-481f-8cf2-14082d3fffd0","Type":"ContainerStarted","Data":"aaaa45078d6439951ca2a5f53127ccefa6b3091b0791c0f0aac9aeb7960cf13c"} Jul 27 21:45:26.344324 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:26.344226 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs\") pod \"network-metrics-daemon-p79rn\" (UID: \"bf30d435-7854-4c24-b72a-c6992b0776a7\") " pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:26.344484 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:26.344431 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:26.344537 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:26.344496 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs podName:bf30d435-7854-4c24-b72a-c6992b0776a7 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:28.344477496 +0000 UTC m=+5.961914799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs") pod "network-metrics-daemon-p79rn" (UID: "bf30d435-7854-4c24-b72a-c6992b0776a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:26.449703 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:26.445118 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:26.449703 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:26.445320 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:26.449703 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:26.445393 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret podName:1ef19975-ecaa-49ba-9cd9-e16737e46c47 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:28.44537609 +0000 UTC m=+6.062813402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret") pod "global-pull-secret-syncer-vlh9m" (UID: "1ef19975-ecaa-49ba-9cd9-e16737e46c47") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:26.546530 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:26.546495 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692z2\" (UniqueName: \"kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2\") pod \"network-check-target-w86pt\" (UID: \"5bdac9f5-97c0-4a7f-9ec7-dd07f0343253\") " pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:26.546708 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:26.546662 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:45:26.546708 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:26.546683 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:45:26.546708 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:26.546697 2561 projected.go:196] Error preparing data for projected volume kube-api-access-692z2 for pod openshift-network-diagnostics/network-check-target-w86pt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:26.546887 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:26.546758 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2 podName:5bdac9f5-97c0-4a7f-9ec7-dd07f0343253 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:28.546739873 +0000 UTC m=+6.164177176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-692z2" (UniqueName: "kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2") pod "network-check-target-w86pt" (UID: "5bdac9f5-97c0-4a7f-9ec7-dd07f0343253") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:26.761106 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:26.761061 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 21:40:24 +0000 UTC" deadline="2028-01-29 17:37:24.866158329 +0000 UTC" Jul 27 21:45:26.761106 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:26.761103 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13219h51m58.105059482s" Jul 27 21:45:26.819884 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:26.819849 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:26.820030 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:26.819991 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:26.820436 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:26.820414 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:26.820537 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:26.820510 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:26.820619 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:26.820567 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:26.820678 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:26.820643 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:26.846393 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:26.846341 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" event={"ID":"f186ee96a5b10264dea34206a4faa43e","Type":"ContainerStarted","Data":"9433c709265cc01268605046b6a541aecef7711484d40a1ad3ed13f355f953af"} Jul 27 21:45:26.855316 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:26.855032 2561 generic.go:356] "Generic (PLEG): container finished" podID="0483318e6ca30ead09c5152f256a0232" containerID="4fef3b28fa9c4adce14b769eb05e7e2325bb405ec25f0bc5256192f3dc5e9df6" exitCode=0 Jul 27 21:45:26.855316 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:26.855086 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" event={"ID":"0483318e6ca30ead09c5152f256a0232","Type":"ContainerDied","Data":"4fef3b28fa9c4adce14b769eb05e7e2325bb405ec25f0bc5256192f3dc5e9df6"} Jul 27 21:45:26.883729 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:26.883651 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" podStartSLOduration=3.8836238119999997 podStartE2EDuration="3.883623812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:26.859416787 +0000 UTC m=+4.476854120" watchObservedRunningTime="2026-07-27 21:45:26.883623812 +0000 UTC m=+4.501061144" Jul 27 21:45:27.881544 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:27.881323 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" event={"ID":"0483318e6ca30ead09c5152f256a0232","Type":"ContainerStarted","Data":"b09be58701aca18d2081118feb63c0679f7e066f32a27430c83f935ef69e57ff"} Jul 27 21:45:27.896816 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:27.896754 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" podStartSLOduration=4.896732214 podStartE2EDuration="4.896732214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:27.896721079 +0000 UTC m=+5.514158400" watchObservedRunningTime="2026-07-27 21:45:27.896732214 +0000 UTC m=+5.514169536" Jul 27 21:45:28.362306 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:28.362203 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs\") pod \"network-metrics-daemon-p79rn\" (UID: \"bf30d435-7854-4c24-b72a-c6992b0776a7\") " pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:28.362464 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:28.362432 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:28.362532 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:28.362500 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs podName:bf30d435-7854-4c24-b72a-c6992b0776a7 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:32.362479579 +0000 UTC m=+9.979916899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs") pod "network-metrics-daemon-p79rn" (UID: "bf30d435-7854-4c24-b72a-c6992b0776a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:28.463604 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:28.463567 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:28.463781 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:28.463733 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:28.463844 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:28.463798 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret podName:1ef19975-ecaa-49ba-9cd9-e16737e46c47 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:32.463780208 +0000 UTC m=+10.081217511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret") pod "global-pull-secret-syncer-vlh9m" (UID: "1ef19975-ecaa-49ba-9cd9-e16737e46c47") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:28.564671 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:28.564600 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692z2\" (UniqueName: \"kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2\") pod \"network-check-target-w86pt\" (UID: \"5bdac9f5-97c0-4a7f-9ec7-dd07f0343253\") " pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:28.564877 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:28.564837 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:45:28.564877 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:28.564858 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:45:28.564877 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:28.564870 2561 projected.go:196] Error preparing data for projected volume kube-api-access-692z2 for pod openshift-network-diagnostics/network-check-target-w86pt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:28.565048 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:28.564928 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2 podName:5bdac9f5-97c0-4a7f-9ec7-dd07f0343253 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:32.564910374 +0000 UTC m=+10.182347675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-692z2" (UniqueName: "kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2") pod "network-check-target-w86pt" (UID: "5bdac9f5-97c0-4a7f-9ec7-dd07f0343253") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:28.813465 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:28.813383 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:28.813619 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:28.813524 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:28.813876 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:28.813857 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:28.813973 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:28.813955 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:28.814042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:28.814015 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:28.814092 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:28.814080 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:30.812927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:30.812424 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:30.812927 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:30.812584 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:30.812927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:30.812767 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:30.812927 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:30.812853 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:30.813573 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:30.812978 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:30.813573 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:30.813062 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:32.399662 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:32.399629 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs\") pod \"network-metrics-daemon-p79rn\" (UID: \"bf30d435-7854-4c24-b72a-c6992b0776a7\") " pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:32.400138 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:32.399803 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:32.400138 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:32.399880 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs podName:bf30d435-7854-4c24-b72a-c6992b0776a7 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:40.399858703 +0000 UTC m=+18.017296004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs") pod "network-metrics-daemon-p79rn" (UID: "bf30d435-7854-4c24-b72a-c6992b0776a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:32.501515 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:32.500854 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:32.501515 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:32.501078 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:32.501515 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:32.501150 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret podName:1ef19975-ecaa-49ba-9cd9-e16737e46c47 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:40.501130206 +0000 UTC m=+18.118567504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret") pod "global-pull-secret-syncer-vlh9m" (UID: "1ef19975-ecaa-49ba-9cd9-e16737e46c47") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:32.602175 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:32.602137 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692z2\" (UniqueName: \"kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2\") pod \"network-check-target-w86pt\" (UID: \"5bdac9f5-97c0-4a7f-9ec7-dd07f0343253\") " pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:32.602868 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:32.602380 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:45:32.602868 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:32.602407 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:45:32.602868 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:32.602420 2561 projected.go:196] Error preparing data for projected volume kube-api-access-692z2 for pod openshift-network-diagnostics/network-check-target-w86pt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:32.602868 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:32.602490 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2 podName:5bdac9f5-97c0-4a7f-9ec7-dd07f0343253 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:40.602470848 +0000 UTC m=+18.219908163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-692z2" (UniqueName: "kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2") pod "network-check-target-w86pt" (UID: "5bdac9f5-97c0-4a7f-9ec7-dd07f0343253") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:32.814107 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:32.814017 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:32.814282 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:32.814129 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:32.814282 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:32.814151 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:32.814282 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:32.814248 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:32.814412 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:32.814316 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:32.814455 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:32.814412 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:34.813061 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:34.813026 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:34.813758 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:34.813153 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:34.813758 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:34.813207 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:34.813758 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:34.813026 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:34.813758 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:34.813306 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:34.813758 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:34.813409 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:36.812721 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:36.812671 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:36.812721 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:36.812710 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:36.813209 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:36.812679 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:36.813209 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:36.812810 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:36.813209 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:36.812921 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:36.813209 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:36.813008 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:38.813057 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:38.813021 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:38.813570 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:38.813074 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:38.813570 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:38.813027 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:38.813570 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:38.813159 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:38.813570 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:38.813251 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:38.813570 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:38.813341 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:40.462392 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:40.462348 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs\") pod \"network-metrics-daemon-p79rn\" (UID: \"bf30d435-7854-4c24-b72a-c6992b0776a7\") " pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:40.462871 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:40.462483 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:40.462871 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:40.462564 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs podName:bf30d435-7854-4c24-b72a-c6992b0776a7 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:56.462540292 +0000 UTC m=+34.079977592 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs") pod "network-metrics-daemon-p79rn" (UID: "bf30d435-7854-4c24-b72a-c6992b0776a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:40.563222 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:40.563140 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:40.563390 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:40.563318 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:40.563465 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:40.563400 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret podName:1ef19975-ecaa-49ba-9cd9-e16737e46c47 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:56.563380253 +0000 UTC m=+34.180817552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret") pod "global-pull-secret-syncer-vlh9m" (UID: "1ef19975-ecaa-49ba-9cd9-e16737e46c47") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:40.664016 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:40.663976 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692z2\" (UniqueName: \"kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2\") pod \"network-check-target-w86pt\" (UID: \"5bdac9f5-97c0-4a7f-9ec7-dd07f0343253\") " pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:40.664201 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:40.664155 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:45:40.664201 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:40.664181 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:45:40.664201 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:40.664197 2561 projected.go:196] Error preparing data for projected volume kube-api-access-692z2 for pod openshift-network-diagnostics/network-check-target-w86pt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:40.664343 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:40.664276 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2 podName:5bdac9f5-97c0-4a7f-9ec7-dd07f0343253 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:56.664240374 +0000 UTC m=+34.281677685 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-692z2" (UniqueName: "kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2") pod "network-check-target-w86pt" (UID: "5bdac9f5-97c0-4a7f-9ec7-dd07f0343253") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:40.812779 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:40.812741 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:40.812779 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:40.812766 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:40.812989 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:40.812892 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:40.813123 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:40.813090 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:40.813281 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:40.813206 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:40.813349 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:40.813316 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:42.813918 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:42.813886 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:42.814404 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:42.814078 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:42.814404 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:42.814152 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:42.814404 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:42.814181 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:42.814404 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:42.814247 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:42.814404 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:42.814043 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:44.814859 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.814835 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:44.815309 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:44.814989 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:44.815446 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.815429 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:44.815544 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:44.815526 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:44.815606 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.815591 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:44.815689 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:44.815673 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:44.919884 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.919633 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" event={"ID":"df4b987a-623d-4079-bbb9-b38e0fa986f5","Type":"ContainerStarted","Data":"8f86f517a6aa13fa80632529ac755003b44eaf447bf5e668d56b0ef7e7d173ae"} Jul 27 21:45:44.922087 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.922059 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lnz2" event={"ID":"4eb68d22-0e6c-4291-9657-fa19b31b6cd6","Type":"ContainerStarted","Data":"b63f123c860b2f8fd4a95d30d3ef351492c3320f71d5a0367a8efd02a9b7a565"} Jul 27 21:45:44.923978 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.923954 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ghrhz" event={"ID":"8aa2c150-1b16-43c8-81a7-a290299036d3","Type":"ContainerStarted","Data":"ffd174882aee4610544894709eb7985b46cd42234bf3e1978193cbfea9400cd8"} Jul 27 21:45:44.925417 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.925398 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6hjdx" event={"ID":"6aaee722-f9e3-4563-addf-2a4fe9f5f0a0","Type":"ContainerStarted","Data":"c61167702c0b01ce4cef94ec943299c8805ba7e3f2bb89593c99f2d06b0d2640"} Jul 27 21:45:44.927073 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.927051 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" event={"ID":"a847c2b0-e057-481f-8cf2-14082d3fffd0","Type":"ContainerStarted","Data":"1501fbc873be1a1c0db7e14f99268b268ed523316a8b7f4b853fd025840fabe9"} Jul 27 21:45:44.928219 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.928189 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8m8z2" event={"ID":"7f949673-96be-4e7f-9cf8-5f69124a6309","Type":"ContainerStarted","Data":"9fd4d7ae874f926b74871f364647a6c9368469fa2a086f1a561044155636a641"} Jul 27 21:45:44.929464 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.929438 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gqtgz" event={"ID":"f919d24f-2d1e-4b16-b946-342af47fe64c","Type":"ContainerStarted","Data":"a60d447e6d4f42f1cb5b461658eb9cabacd977935ba5a3331a3312ac8b6be509"} Jul 27 21:45:44.930764 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.930745 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" event={"ID":"97dc9818-711d-4003-8f27-7bde8cf8176b","Type":"ContainerStarted","Data":"08f973330104c25e92e262e60f24a48b94066486e29e76abadbe11e77f6e60af"} Jul 27 21:45:44.961723 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.961668 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gqtgz" podStartSLOduration=4.069131627 podStartE2EDuration="22.961647363s" totalImagesPullingTime="18.892515736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:44.961304796 +0000 UTC m=+22.578742116" watchObservedRunningTime="2026-07-27 21:45:44.961647363 +0000 UTC m=+22.579084682" Jul 27 21:45:44.973557 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.973517 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6hjdx" podStartSLOduration=3.13764468 podStartE2EDuration="21.973500521s" totalImagesPullingTime="18.835855841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:44.973190698 +0000 UTC m=+22.590628016" watchObservedRunningTime="2026-07-27 21:45:44.973500521 +0000 UTC m=+22.590937839" Jul 27 21:45:44.986653 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:44.986606 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ghrhz" podStartSLOduration=4.136957956 podStartE2EDuration="22.986591975s" totalImagesPullingTime="18.849634019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:44.986114859 +0000 UTC m=+22.603552178" watchObservedRunningTime="2026-07-27 21:45:44.986591975 +0000 UTC m=+22.604029294" Jul 27 21:45:45.001620 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:45.001579 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h9t4q" podStartSLOduration=4.120448607 podStartE2EDuration="23.00156382s" totalImagesPullingTime="18.881115213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:45.001111651 +0000 UTC m=+22.618548991" watchObservedRunningTime="2026-07-27 21:45:45.00156382 +0000 UTC m=+22.619001201" Jul 27 21:45:45.015110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:45.015061 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8m8z2" podStartSLOduration=4.135791492 podStartE2EDuration="23.015041656s" totalImagesPullingTime="18.879250164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:45.01481448 +0000 UTC m=+22.632251800" watchObservedRunningTime="2026-07-27 21:45:45.015041656 +0000 UTC m=+22.632478976" Jul 27 21:45:45.935550 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:45.935247 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m2lx2" event={"ID":"e6c2bede-f097-4c04-8a7b-1c1e83d9d284","Type":"ContainerStarted","Data":"a857585ce8845e60b184997da399a11015ca4f00ecddbe073fb95fb9d2f33f52"} Jul 27 21:45:45.937970 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:45.937942 2561 generic.go:356] "Generic (PLEG): container finished" podID="4eb68d22-0e6c-4291-9657-fa19b31b6cd6" containerID="b63f123c860b2f8fd4a95d30d3ef351492c3320f71d5a0367a8efd02a9b7a565" exitCode=0 Jul 27 21:45:45.938577 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:45.938304 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lnz2" event={"ID":"4eb68d22-0e6c-4291-9657-fa19b31b6cd6","Type":"ContainerDied","Data":"b63f123c860b2f8fd4a95d30d3ef351492c3320f71d5a0367a8efd02a9b7a565"} Jul 27 21:45:45.941889 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:45.941867 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" event={"ID":"a847c2b0-e057-481f-8cf2-14082d3fffd0","Type":"ContainerStarted","Data":"5afc70e81329e0e6310d55a9489df452d5ab8ba182653508346d4d98155222c1"} Jul 27 21:45:45.941980 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:45.941944 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" event={"ID":"a847c2b0-e057-481f-8cf2-14082d3fffd0","Type":"ContainerStarted","Data":"11a8712f63ed25a87a0d98e5226edbd080917d85be1760d0c3b6056874b2868f"} Jul 27 21:45:45.942067 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:45.941999 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" event={"ID":"a847c2b0-e057-481f-8cf2-14082d3fffd0","Type":"ContainerStarted","Data":"1b5085bc46888f856fb7c146238f1ab6e40a7f3bc0b2b4deda6514b15a0e9773"} Jul 27 21:45:45.942067 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:45.942049 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" event={"ID":"a847c2b0-e057-481f-8cf2-14082d3fffd0","Type":"ContainerStarted","Data":"7d732655801c2723303c5275a3797aeb510388821ae3abbb800f688ce3157878"} Jul 27 21:45:45.942158 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:45.942098 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" event={"ID":"a847c2b0-e057-481f-8cf2-14082d3fffd0","Type":"ContainerStarted","Data":"257a9eecd264bc52d46b3821334214425111f4471516e05d3ab7be69825a0a7f"} Jul 27 21:45:45.948958 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:45.948923 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m2lx2" podStartSLOduration=5.071048753 podStartE2EDuration="23.948912436s" totalImagesPullingTime="18.877863683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:45.948567699 +0000 UTC m=+23.566005001" watchObservedRunningTime="2026-07-27 21:45:45.948912436 +0000 UTC m=+23.566349793" Jul 27 21:45:46.060253 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:46.060230 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 21:45:46.793641 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:46.793334 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T21:45:46.060249402Z","UUID":"edcc4e0b-1c10-4a85-9fe5-6b13c3253295","Handler":null,"Name":"","Endpoint":""} Jul 27 21:45:46.795531 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:46.795506 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 21:45:46.795661 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:46.795546 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 21:45:46.812397 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:46.812360 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:46.812583 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:46.812361 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:46.812652 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:46.812606 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:46.812652 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:46.812475 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:46.812652 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:46.812360 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:46.812822 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:46.812746 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:46.946348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:46.946307 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" event={"ID":"df4b987a-623d-4079-bbb9-b38e0fa986f5","Type":"ContainerStarted","Data":"d49d455186d819013af090d3d1735176c76939f05238fe334120e52a784d733a"} Jul 27 21:45:47.951085 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:47.951044 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" event={"ID":"df4b987a-623d-4079-bbb9-b38e0fa986f5","Type":"ContainerStarted","Data":"79ae27ea7e91121d8ad503ab3da2b2e4e2908198621904b83a9a64c1959af13a"} Jul 27 21:45:47.954515 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:47.954488 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" event={"ID":"a847c2b0-e057-481f-8cf2-14082d3fffd0","Type":"ContainerStarted","Data":"f35ec8ce597d886b67bda7c33760200df119df364f0a013c914b9590556c06f6"} Jul 27 21:45:47.967051 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:47.967006 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f42cs" podStartSLOduration=4.964355636 podStartE2EDuration="25.966994983s" totalImagesPullingTime="21.002639347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:47.96620852 +0000 UTC m=+25.583645839" watchObservedRunningTime="2026-07-27 21:45:47.966994983 +0000 UTC m=+25.584432299" Jul 27 21:45:48.812677 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:48.812547 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:48.812871 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:48.812685 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:48.812871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:48.812789 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:48.812982 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:48.812908 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:48.813022 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:48.812974 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:48.813085 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:48.813058 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:49.559244 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:49.559046 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6hjdx" Jul 27 21:45:49.559718 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:49.559650 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6hjdx" Jul 27 21:45:50.813161 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:50.812917 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:50.813892 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:50.812918 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:50.813892 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:50.813292 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:50.813892 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:50.812932 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:50.813892 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:50.813327 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:50.813892 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:50.813413 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:50.962885 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:50.962855 2561 generic.go:356] "Generic (PLEG): container finished" podID="4eb68d22-0e6c-4291-9657-fa19b31b6cd6" containerID="884b673ef315f940c36c0d86a608cf0140f361c76bbcfd635c12e2ab4deb9a1c" exitCode=0 Jul 27 21:45:50.963058 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:50.962948 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lnz2" event={"ID":"4eb68d22-0e6c-4291-9657-fa19b31b6cd6","Type":"ContainerDied","Data":"884b673ef315f940c36c0d86a608cf0140f361c76bbcfd635c12e2ab4deb9a1c"} Jul 27 21:45:50.966684 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:50.966654 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" event={"ID":"a847c2b0-e057-481f-8cf2-14082d3fffd0","Type":"ContainerStarted","Data":"759e69dde162995517af15eaa006263daff8942063f871e2cf10af0f3fead7da"} Jul 27 21:45:50.966974 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:50.966957 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:50.967066 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:50.967026 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:50.982718 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:50.982686 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:50.982862 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:50.982818 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:51.006398 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:51.006348 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" podStartSLOduration=9.954778775 podStartE2EDuration="29.006331066s" totalImagesPullingTime="19.051552291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:51.005219016 +0000 UTC m=+28.622656348" watchObservedRunningTime="2026-07-27 21:45:51.006331066 +0000 UTC m=+28.623768382" Jul 27 21:45:51.968673 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:51.968647 2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 27 21:45:52.251649 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:52.251610 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:52.333933 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:52.333907 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:52.371885 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:52.371823 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vlh9m"] Jul 27 21:45:52.372098 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:52.371953 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:52.372098 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:52.372067 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:52.372592 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:52.372566 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w86pt"] Jul 27 21:45:52.372679 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:52.372654 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:52.372744 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:52.372727 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:52.383578 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:52.383548 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p79rn"] Jul 27 21:45:52.383712 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:52.383641 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:52.383791 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:52.383744 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:52.972408 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:52.972370 2561 generic.go:356] "Generic (PLEG): container finished" podID="4eb68d22-0e6c-4291-9657-fa19b31b6cd6" containerID="4e3cfd05946c0b4237a0c4fde65b52ea017c007cc9aef065b67a291d48226611" exitCode=0 Jul 27 21:45:52.972878 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:52.972452 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lnz2" event={"ID":"4eb68d22-0e6c-4291-9657-fa19b31b6cd6","Type":"ContainerDied","Data":"4e3cfd05946c0b4237a0c4fde65b52ea017c007cc9aef065b67a291d48226611"} Jul 27 21:45:52.972878 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:52.972634 2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 27 21:45:53.813086 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:53.813051 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:53.813288 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:53.813059 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:53.813288 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:53.813165 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:53.813288 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:53.813235 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:53.879041 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:53.879010 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6hjdx" Jul 27 21:45:53.879205 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:53.879123 2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 27 21:45:53.879567 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:53.879546 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6hjdx" Jul 27 21:45:53.901116 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:53.901085 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:53.968626 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:53.968556 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:54.498891 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:54.498711 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:45:54.813115 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:54.813030 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:54.813299 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:54.813137 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:54.978850 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:54.978814 2561 generic.go:356] "Generic (PLEG): container finished" podID="4eb68d22-0e6c-4291-9657-fa19b31b6cd6" containerID="5c59ec3694af5caa69034f50b89c43ac40254e83a4d3b47644834b5c0c4ab738" exitCode=0 Jul 27 21:45:54.979009 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:54.978874 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lnz2" event={"ID":"4eb68d22-0e6c-4291-9657-fa19b31b6cd6","Type":"ContainerDied","Data":"5c59ec3694af5caa69034f50b89c43ac40254e83a4d3b47644834b5c0c4ab738"} Jul 27 21:45:55.138562 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:55.138485 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:55.243379 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:55.243349 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:55.295001 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:55.294970 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:55.812924 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:55.812891 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:55.813476 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:55.812891 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:55.813476 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:55.813019 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w86pt" podUID="5bdac9f5-97c0-4a7f-9ec7-dd07f0343253" Jul 27 21:45:55.813476 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:55.813138 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p79rn" podUID="bf30d435-7854-4c24-b72a-c6992b0776a7" Jul 27 21:45:55.940606 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:55.940521 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:55.995547 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:55.995513 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:56.479153 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:56.479122 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs\") pod \"network-metrics-daemon-p79rn\" (UID: \"bf30d435-7854-4c24-b72a-c6992b0776a7\") " pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:56.479379 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:56.479360 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:56.479456 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:56.479438 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs podName:bf30d435-7854-4c24-b72a-c6992b0776a7 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:28.479419344 +0000 UTC m=+66.096856642 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs") pod "network-metrics-daemon-p79rn" (UID: "bf30d435-7854-4c24-b72a-c6992b0776a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:45:56.580155 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:56.580126 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:56.580348 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:56.580296 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:56.580401 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:56.580366 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret podName:1ef19975-ecaa-49ba-9cd9-e16737e46c47 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:28.580349586 +0000 UTC m=+66.197786882 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret") pod "global-pull-secret-syncer-vlh9m" (UID: "1ef19975-ecaa-49ba-9cd9-e16737e46c47") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:45:56.680891 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:56.680857 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692z2\" (UniqueName: \"kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2\") pod \"network-check-target-w86pt\" (UID: \"5bdac9f5-97c0-4a7f-9ec7-dd07f0343253\") " pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:56.681048 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:56.681021 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:45:56.681048 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:56.681044 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:45:56.681165 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:56.681058 2561 projected.go:196] Error preparing data for projected volume kube-api-access-692z2 for pod openshift-network-diagnostics/network-check-target-w86pt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:56.681165 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:56.681120 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2 podName:5bdac9f5-97c0-4a7f-9ec7-dd07f0343253 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:28.681101173 +0000 UTC m=+66.298538472 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-692z2" (UniqueName: "kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2") pod "network-check-target-w86pt" (UID: "5bdac9f5-97c0-4a7f-9ec7-dd07f0343253") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:45:56.812408 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:56.812322 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:56.812570 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:56.812437 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlh9m" podUID="1ef19975-ecaa-49ba-9cd9-e16737e46c47" Jul 27 21:45:57.174002 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.173979 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeReady" Jul 27 21:45:57.174517 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.174113 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 21:45:57.208363 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.208336 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-986d7665c-s5xvq"] Jul 27 21:45:57.210366 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.210346 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.212674 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.212652 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8f42l\"" Jul 27 21:45:57.212778 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.212654 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 27 21:45:57.213050 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.213033 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 27 21:45:57.213319 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.213296 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 27 21:45:57.218851 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.218663 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 27 21:45:57.219471 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.219443 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z2qwk"] Jul 27 21:45:57.223561 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.223540 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c77xc"] Jul 27 21:45:57.223678 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.223662 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z2qwk" Jul 27 21:45:57.225423 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.225401 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-986d7665c-s5xvq"] Jul 27 21:45:57.225516 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.225495 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:57.226339 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.226188 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 21:45:57.226339 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.226192 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 21:45:57.226580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.226564 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 21:45:57.226669 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.226649 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b6tfk\"" Jul 27 21:45:57.227643 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.227625 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 21:45:57.227734 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.227642 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 21:45:57.227864 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.227851 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jb57h\"" Jul 27 21:45:57.231050 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.231026 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z2qwk"] Jul 27 21:45:57.236034 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.235997 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c77xc"] Jul 27 21:45:57.333648 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.333613 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:57.386081 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386050 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:57.386238 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386094 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsznf\" (UniqueName: \"kubernetes.io/projected/8529dd01-316a-4c4b-af14-e04511a99734-kube-api-access-zsznf\") pod \"ingress-canary-z2qwk\" (UID: \"8529dd01-316a-4c4b-af14-e04511a99734\") " pod="openshift-ingress-canary/ingress-canary-z2qwk" Jul 27 21:45:57.386238 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386117 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-bound-sa-token\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.386238 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386147 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.386238 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386164 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:57.386238 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386221 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cd55b06-4d11-468a-9991-473beba0001b-ca-trust-extracted\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.386509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386278 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pdbd\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-kube-api-access-9pdbd\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.386509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386312 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c159765-958d-4936-a11a-092a2566089b-tmp-dir\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:57.386509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386342 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cd55b06-4d11-468a-9991-473beba0001b-installation-pull-secrets\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.386509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386367 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnf8c\" (UniqueName: \"kubernetes.io/projected/3c159765-958d-4936-a11a-092a2566089b-kube-api-access-xnf8c\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:57.386509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386395 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cd55b06-4d11-468a-9991-473beba0001b-image-registry-private-configuration\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.386509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386417 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cd55b06-4d11-468a-9991-473beba0001b-registry-certificates\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.386509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386439 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8529dd01-316a-4c4b-af14-e04511a99734-cert\") pod \"ingress-canary-z2qwk\" (UID: \"8529dd01-316a-4c4b-af14-e04511a99734\") " pod="openshift-ingress-canary/ingress-canary-z2qwk" Jul 27 21:45:57.386509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386462 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cd55b06-4d11-468a-9991-473beba0001b-trusted-ca\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.386509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.386488 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c159765-958d-4936-a11a-092a2566089b-config-volume\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:57.487418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487336 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cd55b06-4d11-468a-9991-473beba0001b-ca-trust-extracted\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.487418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487384 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdbd\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-kube-api-access-9pdbd\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.487418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487416 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c159765-958d-4936-a11a-092a2566089b-tmp-dir\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:57.487666 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487448 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cd55b06-4d11-468a-9991-473beba0001b-installation-pull-secrets\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.487666 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487476 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnf8c\" (UniqueName: \"kubernetes.io/projected/3c159765-958d-4936-a11a-092a2566089b-kube-api-access-xnf8c\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:57.487666 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487503 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cd55b06-4d11-468a-9991-473beba0001b-image-registry-private-configuration\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.487666 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487520 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cd55b06-4d11-468a-9991-473beba0001b-registry-certificates\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.487666 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487541 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8529dd01-316a-4c4b-af14-e04511a99734-cert\") pod \"ingress-canary-z2qwk\" (UID: \"8529dd01-316a-4c4b-af14-e04511a99734\") " pod="openshift-ingress-canary/ingress-canary-z2qwk" Jul 27 21:45:57.487666 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487570 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cd55b06-4d11-468a-9991-473beba0001b-trusted-ca\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.487666 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487592 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c159765-958d-4936-a11a-092a2566089b-config-volume\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:57.487666 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487659 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsznf\" (UniqueName: \"kubernetes.io/projected/8529dd01-316a-4c4b-af14-e04511a99734-kube-api-access-zsznf\") pod \"ingress-canary-z2qwk\" (UID: \"8529dd01-316a-4c4b-af14-e04511a99734\") " pod="openshift-ingress-canary/ingress-canary-z2qwk" Jul 27 21:45:57.488039 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487688 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-bound-sa-token\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.488039 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487722 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.488039 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487747 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:57.488039 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487804 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c159765-958d-4936-a11a-092a2566089b-tmp-dir\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:57.488039 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.487804 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cd55b06-4d11-468a-9991-473beba0001b-ca-trust-extracted\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.488039 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:57.487850 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:45:57.488039 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:57.487901 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls podName:3c159765-958d-4936-a11a-092a2566089b nodeName:}" failed. No retries permitted until 2026-07-27 21:45:57.987883489 +0000 UTC m=+35.605320786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls") pod "dns-default-c77xc" (UID: "3c159765-958d-4936-a11a-092a2566089b") : secret "dns-default-metrics-tls" not found Jul 27 21:45:57.488401 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:57.488154 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:45:57.488401 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:57.488167 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-986d7665c-s5xvq: secret "image-registry-tls" not found Jul 27 21:45:57.488401 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:57.488205 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls podName:9cd55b06-4d11-468a-9991-473beba0001b nodeName:}" failed. No retries permitted until 2026-07-27 21:45:57.988192584 +0000 UTC m=+35.605629881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls") pod "image-registry-986d7665c-s5xvq" (UID: "9cd55b06-4d11-468a-9991-473beba0001b") : secret "image-registry-tls" not found Jul 27 21:45:57.488401 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.488359 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cd55b06-4d11-468a-9991-473beba0001b-registry-certificates\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.488634 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.488611 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cd55b06-4d11-468a-9991-473beba0001b-trusted-ca\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.488705 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.488686 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c159765-958d-4936-a11a-092a2566089b-config-volume\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:57.492321 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.492300 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cd55b06-4d11-468a-9991-473beba0001b-installation-pull-secrets\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.492321 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.492312 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cd55b06-4d11-468a-9991-473beba0001b-image-registry-private-configuration\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.492888 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.492868 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8529dd01-316a-4c4b-af14-e04511a99734-cert\") pod \"ingress-canary-z2qwk\" (UID: \"8529dd01-316a-4c4b-af14-e04511a99734\") " pod="openshift-ingress-canary/ingress-canary-z2qwk" Jul 27 21:45:57.496651 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.496626 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pdbd\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-kube-api-access-9pdbd\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.496752 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.496676 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnf8c\" (UniqueName: \"kubernetes.io/projected/3c159765-958d-4936-a11a-092a2566089b-kube-api-access-xnf8c\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:57.497639 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.497616 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsznf\" (UniqueName: \"kubernetes.io/projected/8529dd01-316a-4c4b-af14-e04511a99734-kube-api-access-zsznf\") pod \"ingress-canary-z2qwk\" (UID: \"8529dd01-316a-4c4b-af14-e04511a99734\") " pod="openshift-ingress-canary/ingress-canary-z2qwk" Jul 27 21:45:57.497944 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.497923 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-bound-sa-token\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.537194 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.537149 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z2qwk" Jul 27 21:45:57.812363 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.812282 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:45:57.812532 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.812281 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:45:57.814586 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.814564 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cj7gv\"" Jul 27 21:45:57.815508 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.815491 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kt796\"" Jul 27 21:45:57.815619 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.815519 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 21:45:57.815619 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.815535 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 21:45:57.815619 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.815523 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 21:45:57.991376 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.991306 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:57.991376 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:57.991380 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:57.991586 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:57.991474 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:45:57.991586 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:57.991491 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-986d7665c-s5xvq: secret "image-registry-tls" not found Jul 27 21:45:57.991586 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:57.991513 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:45:57.991586 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:57.991565 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls podName:9cd55b06-4d11-468a-9991-473beba0001b nodeName:}" failed. No retries permitted until 2026-07-27 21:45:58.991546647 +0000 UTC m=+36.608983957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls") pod "image-registry-986d7665c-s5xvq" (UID: "9cd55b06-4d11-468a-9991-473beba0001b") : secret "image-registry-tls" not found Jul 27 21:45:57.991586 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:57.991578 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls podName:3c159765-958d-4936-a11a-092a2566089b nodeName:}" failed. No retries permitted until 2026-07-27 21:45:58.991572664 +0000 UTC m=+36.609009960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls") pod "dns-default-c77xc" (UID: "3c159765-958d-4936-a11a-092a2566089b") : secret "dns-default-metrics-tls" not found Jul 27 21:45:58.220558 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:58.220467 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:58.321957 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:58.321919 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:58.378814 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:58.378771 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:58.812816 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:58.812787 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:45:58.815293 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:58.815253 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 21:45:58.974600 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:58.974435 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z2qwk"] Jul 27 21:45:58.978644 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:45:58.978598 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8529dd01_316a_4c4b_af14_e04511a99734.slice/crio-1131de7392020355b225e7fbfbdd53448445d84037500586138e836b8362aff1 WatchSource:0}: Error finding container 1131de7392020355b225e7fbfbdd53448445d84037500586138e836b8362aff1: Status 404 returned error can't find the container with id 1131de7392020355b225e7fbfbdd53448445d84037500586138e836b8362aff1 Jul 27 21:45:58.993353 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:58.993314 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z2qwk" event={"ID":"8529dd01-316a-4c4b-af14-e04511a99734","Type":"ContainerStarted","Data":"1131de7392020355b225e7fbfbdd53448445d84037500586138e836b8362aff1"} Jul 27 21:45:58.996250 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:58.996217 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lnz2" event={"ID":"4eb68d22-0e6c-4291-9657-fa19b31b6cd6","Type":"ContainerStarted","Data":"29b20e2fa2f16f6c6a3b53a443f5e8db37f33a38ade04d58842d4fff901693cf"} Jul 27 21:45:59.001736 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:59.001711 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:45:59.001836 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:59.001745 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:45:59.001888 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:59.001849 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:45:59.001888 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:59.001860 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:45:59.001888 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:59.001879 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-986d7665c-s5xvq: secret "image-registry-tls" not found Jul 27 21:45:59.001989 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:59.001931 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls podName:3c159765-958d-4936-a11a-092a2566089b nodeName:}" failed. No retries permitted until 2026-07-27 21:46:01.00191358 +0000 UTC m=+38.619350876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls") pod "dns-default-c77xc" (UID: "3c159765-958d-4936-a11a-092a2566089b") : secret "dns-default-metrics-tls" not found Jul 27 21:45:59.001989 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:45:59.001944 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls podName:9cd55b06-4d11-468a-9991-473beba0001b nodeName:}" failed. No retries permitted until 2026-07-27 21:46:01.001937866 +0000 UTC m=+38.619375163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls") pod "image-registry-986d7665c-s5xvq" (UID: "9cd55b06-4d11-468a-9991-473beba0001b") : secret "image-registry-tls" not found Jul 27 21:45:59.250547 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:59.250515 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:59.303281 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:45:59.303231 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:00.001569 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:00.001539 2561 generic.go:356] "Generic (PLEG): container finished" podID="4eb68d22-0e6c-4291-9657-fa19b31b6cd6" containerID="29b20e2fa2f16f6c6a3b53a443f5e8db37f33a38ade04d58842d4fff901693cf" exitCode=0 Jul 27 21:46:00.001753 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:00.001597 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lnz2" event={"ID":"4eb68d22-0e6c-4291-9657-fa19b31b6cd6","Type":"ContainerDied","Data":"29b20e2fa2f16f6c6a3b53a443f5e8db37f33a38ade04d58842d4fff901693cf"} Jul 27 21:46:00.998342 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:00.998308 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:01.007695 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.007665 2561 generic.go:356] "Generic (PLEG): container finished" podID="4eb68d22-0e6c-4291-9657-fa19b31b6cd6" containerID="acb7db72083f58e4b5e90be9dd4e4d5bf7fba5289494b903fbcfeb55f0c7320b" exitCode=0 Jul 27 21:46:01.007859 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.007710 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lnz2" event={"ID":"4eb68d22-0e6c-4291-9657-fa19b31b6cd6","Type":"ContainerDied","Data":"acb7db72083f58e4b5e90be9dd4e4d5bf7fba5289494b903fbcfeb55f0c7320b"} Jul 27 21:46:01.016917 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.016881 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:46:01.017132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.016921 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:46:01.017132 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:01.017031 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:46:01.017132 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:01.017051 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-986d7665c-s5xvq: secret "image-registry-tls" not found Jul 27 21:46:01.017132 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:01.017081 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:46:01.017132 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:01.017119 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls podName:9cd55b06-4d11-468a-9991-473beba0001b nodeName:}" failed. No retries permitted until 2026-07-27 21:46:05.017096207 +0000 UTC m=+42.634533505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls") pod "image-registry-986d7665c-s5xvq" (UID: "9cd55b06-4d11-468a-9991-473beba0001b") : secret "image-registry-tls" not found Jul 27 21:46:01.017427 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:01.017150 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls podName:3c159765-958d-4936-a11a-092a2566089b nodeName:}" failed. No retries permitted until 2026-07-27 21:46:05.017135288 +0000 UTC m=+42.634572585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls") pod "dns-default-c77xc" (UID: "3c159765-958d-4936-a11a-092a2566089b") : secret "dns-default-metrics-tls" not found Jul 27 21:46:01.018329 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.018310 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-82g59"] Jul 27 21:46:01.057191 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.057156 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-82g59"] Jul 27 21:46:01.057382 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.057299 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-82g59" Jul 27 21:46:01.059662 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.059638 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 27 21:46:01.059900 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.059882 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 27 21:46:01.060140 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.060109 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-prb2x\"" Jul 27 21:46:01.094478 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.094426 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:01.155200 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.155170 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:01.217037 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.217008 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:01.217873 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.217851 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6lv4\" (UniqueName: \"kubernetes.io/projected/f03b7af1-4768-402b-8b41-f2c48e1a51db-kube-api-access-x6lv4\") pod \"migrator-6865c9967-82g59\" (UID: \"f03b7af1-4768-402b-8b41-f2c48e1a51db\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-82g59" Jul 27 21:46:01.318670 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.318501 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lv4\" (UniqueName: \"kubernetes.io/projected/f03b7af1-4768-402b-8b41-f2c48e1a51db-kube-api-access-x6lv4\") pod \"migrator-6865c9967-82g59\" (UID: \"f03b7af1-4768-402b-8b41-f2c48e1a51db\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-82g59" Jul 27 21:46:01.327856 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.327824 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6lv4\" (UniqueName: \"kubernetes.io/projected/f03b7af1-4768-402b-8b41-f2c48e1a51db-kube-api-access-x6lv4\") pod \"migrator-6865c9967-82g59\" (UID: \"f03b7af1-4768-402b-8b41-f2c48e1a51db\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-82g59" Jul 27 21:46:01.367731 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.367704 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-82g59" Jul 27 21:46:01.368718 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.368695 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:01.470028 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.470000 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:01.695719 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.695695 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-82g59"] Jul 27 21:46:01.699306 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:46:01.699280 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03b7af1_4768_402b_8b41_f2c48e1a51db.slice/crio-b58f57946627bd9496e754d151dc9ec2e3c5203575b1d170b738e8e069131824 WatchSource:0}: Error finding container b58f57946627bd9496e754d151dc9ec2e3c5203575b1d170b738e8e069131824: Status 404 returned error can't find the container with id b58f57946627bd9496e754d151dc9ec2e3c5203575b1d170b738e8e069131824 Jul 27 21:46:01.804018 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.803967 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:01.853310 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:01.853220 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:02.013102 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:02.013073 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lnz2" event={"ID":"4eb68d22-0e6c-4291-9657-fa19b31b6cd6","Type":"ContainerStarted","Data":"b8485f2efe9dd711f91d7c83e2fda5c4ed481729e4dd853d97dc5e6b8c966ee6"} Jul 27 21:46:02.014568 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:02.014537 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z2qwk" event={"ID":"8529dd01-316a-4c4b-af14-e04511a99734","Type":"ContainerStarted","Data":"d6fb3b491c9de3f6e2800d182316e60adfe748ab052695e29f8d4fe8c02d4c50"} Jul 27 21:46:02.015580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:02.015549 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-82g59" event={"ID":"f03b7af1-4768-402b-8b41-f2c48e1a51db","Type":"ContainerStarted","Data":"b58f57946627bd9496e754d151dc9ec2e3c5203575b1d170b738e8e069131824"} Jul 27 21:46:02.034816 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:02.034754 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9lnz2" podStartSLOduration=10.955171038 podStartE2EDuration="40.034733904s" totalImagesPullingTime="29.079562866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:02.033395156 +0000 UTC m=+39.650832478" watchObservedRunningTime="2026-07-27 21:46:02.034733904 +0000 UTC m=+39.652171224" Jul 27 21:46:02.048566 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:02.048487 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z2qwk" podStartSLOduration=2.456541374 podStartE2EDuration="5.048471714s" totalImagesPullingTime="2.59193034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:02.047892008 +0000 UTC m=+39.665329327" watchObservedRunningTime="2026-07-27 21:46:02.048471714 +0000 UTC m=+39.665909032" Jul 27 21:46:02.164375 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:02.164294 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:02.265185 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:02.265157 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:02.367297 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:02.367245 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:02.859411 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:02.859211 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:02.914940 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:02.914897 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:03.114603 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.114505 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:03.191063 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.191033 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:03.674789 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.674708 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-85h7m"] Jul 27 21:46:03.695010 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.694930 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-85h7m"] Jul 27 21:46:03.695169 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.695014 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" Jul 27 21:46:03.697094 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.697071 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4nzzh\"" Jul 27 21:46:03.697351 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.697336 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 27 21:46:03.697691 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.697665 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 27 21:46:03.697765 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.697736 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 27 21:46:03.697818 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.697735 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 27 21:46:03.712571 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.712538 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:03.837768 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.837732 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdp9s\" (UniqueName: \"kubernetes.io/projected/4bf014cd-90b8-4cff-b425-95e93fb4287f-kube-api-access-bdp9s\") pod \"service-ca-658d7b7c78-85h7m\" (UID: \"4bf014cd-90b8-4cff-b425-95e93fb4287f\") " pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" Jul 27 21:46:03.837930 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.837794 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bf014cd-90b8-4cff-b425-95e93fb4287f-signing-key\") pod \"service-ca-658d7b7c78-85h7m\" (UID: \"4bf014cd-90b8-4cff-b425-95e93fb4287f\") " pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" Jul 27 21:46:03.837930 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.837851 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bf014cd-90b8-4cff-b425-95e93fb4287f-signing-cabundle\") pod \"service-ca-658d7b7c78-85h7m\" (UID: \"4bf014cd-90b8-4cff-b425-95e93fb4287f\") " pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" Jul 27 21:46:03.938421 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.938353 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdp9s\" (UniqueName: \"kubernetes.io/projected/4bf014cd-90b8-4cff-b425-95e93fb4287f-kube-api-access-bdp9s\") pod \"service-ca-658d7b7c78-85h7m\" (UID: \"4bf014cd-90b8-4cff-b425-95e93fb4287f\") " pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" Jul 27 21:46:03.938421 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.938406 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bf014cd-90b8-4cff-b425-95e93fb4287f-signing-key\") pod \"service-ca-658d7b7c78-85h7m\" (UID: \"4bf014cd-90b8-4cff-b425-95e93fb4287f\") " pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" Jul 27 21:46:03.938556 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.938435 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bf014cd-90b8-4cff-b425-95e93fb4287f-signing-cabundle\") pod \"service-ca-658d7b7c78-85h7m\" (UID: \"4bf014cd-90b8-4cff-b425-95e93fb4287f\") " pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" Jul 27 21:46:03.939169 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.939151 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bf014cd-90b8-4cff-b425-95e93fb4287f-signing-cabundle\") pod \"service-ca-658d7b7c78-85h7m\" (UID: \"4bf014cd-90b8-4cff-b425-95e93fb4287f\") " pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" Jul 27 21:46:03.941597 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.941577 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bf014cd-90b8-4cff-b425-95e93fb4287f-signing-key\") pod \"service-ca-658d7b7c78-85h7m\" (UID: \"4bf014cd-90b8-4cff-b425-95e93fb4287f\") " pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" Jul 27 21:46:03.945763 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:03.945741 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdp9s\" (UniqueName: \"kubernetes.io/projected/4bf014cd-90b8-4cff-b425-95e93fb4287f-kube-api-access-bdp9s\") pod \"service-ca-658d7b7c78-85h7m\" (UID: \"4bf014cd-90b8-4cff-b425-95e93fb4287f\") " pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" Jul 27 21:46:04.004325 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:04.004291 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" Jul 27 21:46:04.022030 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:04.022003 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-82g59" event={"ID":"f03b7af1-4768-402b-8b41-f2c48e1a51db","Type":"ContainerStarted","Data":"6f29a96c13aec7e32259b81633e9b21ca1bc5f49f460309b8d645bc074c89a89"} Jul 27 21:46:04.022124 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:04.022060 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-82g59" event={"ID":"f03b7af1-4768-402b-8b41-f2c48e1a51db","Type":"ContainerStarted","Data":"d1e801d00d4459eaad07af2b8ab15b702e134dc2868e6f45407836180385dc08"} Jul 27 21:46:04.038369 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:04.038321 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-82g59" podStartSLOduration=2.393408216 podStartE2EDuration="4.038306419s" totalImagesPullingTime="1.644898203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:04.037716768 +0000 UTC m=+41.655154088" watchObservedRunningTime="2026-07-27 21:46:04.038306419 +0000 UTC m=+41.655743737" Jul 27 21:46:04.114685 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:04.114656 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:04.135871 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:04.135840 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-85h7m"] Jul 27 21:46:04.139753 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:46:04.139727 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf014cd_90b8_4cff_b425_95e93fb4287f.slice/crio-57d49a1f31b8da3b9d9e1727a96154751bf9b26c8bee5b527a5a26db2c7c439c WatchSource:0}: Error finding container 57d49a1f31b8da3b9d9e1727a96154751bf9b26c8bee5b527a5a26db2c7c439c: Status 404 returned error can't find the container with id 57d49a1f31b8da3b9d9e1727a96154751bf9b26c8bee5b527a5a26db2c7c439c Jul 27 21:46:04.198920 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:04.198848 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:04.567070 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:04.566998 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:05.026143 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:05.026072 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" event={"ID":"4bf014cd-90b8-4cff-b425-95e93fb4287f","Type":"ContainerStarted","Data":"57d49a1f31b8da3b9d9e1727a96154751bf9b26c8bee5b527a5a26db2c7c439c"} Jul 27 21:46:05.046042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:05.046009 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:46:05.046042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:05.046049 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:46:05.046288 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:05.046167 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:46:05.046288 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:05.046169 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:46:05.046288 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:05.046189 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-986d7665c-s5xvq: secret "image-registry-tls" not found Jul 27 21:46:05.046288 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:05.046222 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls podName:3c159765-958d-4936-a11a-092a2566089b nodeName:}" failed. No retries permitted until 2026-07-27 21:46:13.046203802 +0000 UTC m=+50.663641113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls") pod "dns-default-c77xc" (UID: "3c159765-958d-4936-a11a-092a2566089b") : secret "dns-default-metrics-tls" not found Jul 27 21:46:05.046288 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:05.046241 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls podName:9cd55b06-4d11-468a-9991-473beba0001b nodeName:}" failed. No retries permitted until 2026-07-27 21:46:13.046231702 +0000 UTC m=+50.663669001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls") pod "image-registry-986d7665c-s5xvq" (UID: "9cd55b06-4d11-468a-9991-473beba0001b") : secret "image-registry-tls" not found Jul 27 21:46:07.031662 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:07.031622 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" event={"ID":"4bf014cd-90b8-4cff-b425-95e93fb4287f","Type":"ContainerStarted","Data":"d6411268394fffb33973cd4d8e3497b6ee748754274b91105ff8473bbfc650af"} Jul 27 21:46:07.055576 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:07.055523 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-85h7m" podStartSLOduration=1.836475614 podStartE2EDuration="4.055504956s" totalImagesPullingTime="2.219029342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:07.054402289 +0000 UTC m=+44.671839610" watchObservedRunningTime="2026-07-27 21:46:07.055504956 +0000 UTC m=+44.672942276" Jul 27 21:46:07.208143 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:07.208098 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:07.412974 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:07.412945 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:07.513519 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:07.513483 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:07.647517 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:07.647483 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:07.747002 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:07.746921 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:07.847913 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:07.847880 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:07.947634 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:07.947601 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:08.924906 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:08.924880 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:09.304649 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:09.304571 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:09.505969 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:09.505933 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:09.703341 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:09.703274 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:09.909300 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:09.909251 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:09.959837 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:09.959757 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:10.912403 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:10.912377 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:11.012536 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:11.012509 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:11.112888 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:11.112854 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:11.505926 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:11.505894 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:11.707191 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:11.707160 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:11.759114 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:11.759034 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:11.913459 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:11.913429 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:12.907072 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:12.907041 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:13.104675 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:13.104638 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:46:13.104675 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:13.104679 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:46:13.106986 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:13.106963 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c159765-958d-4936-a11a-092a2566089b-metrics-tls\") pod \"dns-default-c77xc\" (UID: \"3c159765-958d-4936-a11a-092a2566089b\") " pod="openshift-dns/dns-default-c77xc" Jul 27 21:46:13.107107 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:13.107018 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls\") pod \"image-registry-986d7665c-s5xvq\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:46:13.122941 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:13.122919 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:46:13.142523 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:13.142495 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c77xc" Jul 27 21:46:13.275875 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:13.275838 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-986d7665c-s5xvq"] Jul 27 21:46:13.282596 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:46:13.282561 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd55b06_4d11_468a_9991_473beba0001b.slice/crio-5c714e281091d60b81d8a7a5e8832c6931b1ce51a950a30c3c3f0530251cd8af WatchSource:0}: Error finding container 5c714e281091d60b81d8a7a5e8832c6931b1ce51a950a30c3c3f0530251cd8af: Status 404 returned error can't find the container with id 5c714e281091d60b81d8a7a5e8832c6931b1ce51a950a30c3c3f0530251cd8af Jul 27 21:46:13.309516 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:13.309333 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c77xc"] Jul 27 21:46:13.314049 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:46:13.314025 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c159765_958d_4936_a11a_092a2566089b.slice/crio-169ae80375810a7a1e094f8838ea9283d80ba4aed2a8d2f91c911b54518ad275 WatchSource:0}: Error finding container 169ae80375810a7a1e094f8838ea9283d80ba4aed2a8d2f91c911b54518ad275: Status 404 returned error can't find the container with id 169ae80375810a7a1e094f8838ea9283d80ba4aed2a8d2f91c911b54518ad275 Jul 27 21:46:13.318502 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:13.318473 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:13.418629 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:13.418506 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:13.519711 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:13.519678 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:13.702953 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:13.702843 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:13.912413 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:13.912381 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:14.052198 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:14.051857 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-986d7665c-s5xvq" event={"ID":"9cd55b06-4d11-468a-9991-473beba0001b","Type":"ContainerStarted","Data":"89560eb49f972945beb9970410f1a1d8007d9a7b659e6dab4d5472eb5642b6fc"} Jul 27 21:46:14.052198 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:14.051911 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-986d7665c-s5xvq" event={"ID":"9cd55b06-4d11-468a-9991-473beba0001b","Type":"ContainerStarted","Data":"5c714e281091d60b81d8a7a5e8832c6931b1ce51a950a30c3c3f0530251cd8af"} Jul 27 21:46:14.052198 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:14.052091 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:46:14.053423 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:14.053400 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c77xc" event={"ID":"3c159765-958d-4936-a11a-092a2566089b","Type":"ContainerStarted","Data":"169ae80375810a7a1e094f8838ea9283d80ba4aed2a8d2f91c911b54518ad275"} Jul 27 21:46:14.074764 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:14.074704 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-986d7665c-s5xvq" podStartSLOduration=38.074688569 podStartE2EDuration="38.074688569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:14.073570338 +0000 UTC m=+51.691007660" watchObservedRunningTime="2026-07-27 21:46:14.074688569 +0000 UTC m=+51.692125888" Jul 27 21:46:14.705798 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:14.705770 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:15.110599 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:15.110507 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:15.307405 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:15.307368 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:15.711945 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:15.711916 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:15.850579 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:15.850488 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:15.950903 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:15.950868 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:16.061322 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:16.061283 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c77xc" event={"ID":"3c159765-958d-4936-a11a-092a2566089b","Type":"ContainerStarted","Data":"660479f587b22f3644b3c530d940136f07c43fc0066a30c1dc10a85fdd1ee1c7"} Jul 27 21:46:16.061498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:16.061371 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c77xc" event={"ID":"3c159765-958d-4936-a11a-092a2566089b","Type":"ContainerStarted","Data":"e3baeab18c87fd8c375298e42fd8b48ad317ff02ba2b80986a9a0ce717e39921"} Jul 27 21:46:16.061498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:16.061392 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c77xc" Jul 27 21:46:16.090459 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:16.090414 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c77xc" podStartSLOduration=16.864047757 podStartE2EDuration="19.090398617s" totalImagesPullingTime="2.22635086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:16.089620589 +0000 UTC m=+53.707057908" watchObservedRunningTime="2026-07-27 21:46:16.090398617 +0000 UTC m=+53.707835976" Jul 27 21:46:16.225565 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:16.225533 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:16.326280 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:16.326229 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:16.909488 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:16.909454 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:17.116926 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:17.116898 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:17.316973 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:17.316875 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:17.476481 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:17.476448 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:17.577840 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:17.577733 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:17.637670 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:17.637611 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:17.772625 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:17.772595 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:18.144936 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:18.144901 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:18.203900 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:18.203867 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:18.542066 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:18.542012 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:18.601050 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:18.600989 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:18.740629 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:18.740598 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:18.798453 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:18.798364 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:18.953430 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:18.953402 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:19.129926 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:19.129830 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:19.184569 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:19.184517 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:20.770247 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:20.770215 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:20.820384 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:20.820357 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:23.507500 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.507468 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-986d7665c-s5xvq"] Jul 27 21:46:23.524915 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.524886 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5"] Jul 27 21:46:23.543572 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.543543 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-b8v9l"] Jul 27 21:46:23.543716 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.543663 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" Jul 27 21:46:23.546702 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.546661 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 21:46:23.546702 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.546699 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2kvt2\"" Jul 27 21:46:23.546920 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.546902 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 21:46:23.546979 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.546966 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 27 21:46:23.547032 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.546906 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 27 21:46:23.557140 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.557115 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5b55db7595-rmmgf"] Jul 27 21:46:23.557367 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.557346 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.559739 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.559722 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 27 21:46:23.559971 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.559953 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 21:46:23.560049 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.560036 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vk8cs\"" Jul 27 21:46:23.560219 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.560201 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 21:46:23.560655 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.560636 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 27 21:46:23.567368 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.567347 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 27 21:46:23.573204 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.573184 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5"] Jul 27 21:46:23.573311 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.573217 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-b8v9l"] Jul 27 21:46:23.573311 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.573246 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b55db7595-rmmgf"] Jul 27 21:46:23.573395 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.573334 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.576498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.576479 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 27 21:46:23.576591 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.576549 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 27 21:46:23.576758 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.576747 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 27 21:46:23.576798 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.576757 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 27 21:46:23.576860 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.576839 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s86p\" (UniqueName: \"kubernetes.io/projected/8bc9020b-6a51-4caa-9163-0d474d41b484-kube-api-access-5s86p\") pod \"cluster-monitoring-operator-548b489cb8-l9zg5\" (UID: \"8bc9020b-6a51-4caa-9163-0d474d41b484\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" Jul 27 21:46:23.576922 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.576900 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3596aa9-4bcf-4506-aff0-3706719f73c1-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.577024 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.576931 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bc9020b-6a51-4caa-9163-0d474d41b484-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-l9zg5\" (UID: \"8bc9020b-6a51-4caa-9163-0d474d41b484\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" Jul 27 21:46:23.577024 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.576955 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3596aa9-4bcf-4506-aff0-3706719f73c1-tmp\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.577024 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.576985 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d3596aa9-4bcf-4506-aff0-3706719f73c1-snapshots\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.577024 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.577004 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8bc9020b-6a51-4caa-9163-0d474d41b484-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-l9zg5\" (UID: \"8bc9020b-6a51-4caa-9163-0d474d41b484\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" Jul 27 21:46:23.577024 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.577023 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d3596aa9-4bcf-4506-aff0-3706719f73c1-serving-cert\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.577228 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.577110 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3596aa9-4bcf-4506-aff0-3706719f73c1-service-ca-bundle\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.577228 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.577142 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 21:46:23.577228 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.577162 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt8jk\" (UniqueName: \"kubernetes.io/projected/d3596aa9-4bcf-4506-aff0-3706719f73c1-kube-api-access-pt8jk\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.577379 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.577366 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 21:46:23.577858 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.577844 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fd2r6\"" Jul 27 21:46:23.640082 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.640046 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-774b54654c-9kg49"] Jul 27 21:46:23.660324 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.660298 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2"] Jul 27 21:46:23.660507 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.660439 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.677806 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.677779 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/157ac3ef-1c87-44e5-8c59-edd637bc0c48-trusted-ca\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.677942 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.677831 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s86p\" (UniqueName: \"kubernetes.io/projected/8bc9020b-6a51-4caa-9163-0d474d41b484-kube-api-access-5s86p\") pod \"cluster-monitoring-operator-548b489cb8-l9zg5\" (UID: \"8bc9020b-6a51-4caa-9163-0d474d41b484\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" Jul 27 21:46:23.677942 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.677869 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/157ac3ef-1c87-44e5-8c59-edd637bc0c48-ca-trust-extracted\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.677942 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.677913 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77c6bb84-f632-4621-958c-51b870956328-service-ca-bundle\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.677942 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.677939 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3596aa9-4bcf-4506-aff0-3706719f73c1-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.678117 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.677956 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245s9\" (UniqueName: \"kubernetes.io/projected/77c6bb84-f632-4621-958c-51b870956328-kube-api-access-245s9\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.678117 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.677975 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bc9020b-6a51-4caa-9163-0d474d41b484-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-l9zg5\" (UID: \"8bc9020b-6a51-4caa-9163-0d474d41b484\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" Jul 27 21:46:23.678117 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.677991 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/157ac3ef-1c87-44e5-8c59-edd637bc0c48-registry-tls\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.678117 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678006 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/157ac3ef-1c87-44e5-8c59-edd637bc0c48-registry-certificates\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.678117 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678025 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3596aa9-4bcf-4506-aff0-3706719f73c1-tmp\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.678117 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678067 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d3596aa9-4bcf-4506-aff0-3706719f73c1-snapshots\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.678117 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678098 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8bc9020b-6a51-4caa-9163-0d474d41b484-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-l9zg5\" (UID: \"8bc9020b-6a51-4caa-9163-0d474d41b484\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" Jul 27 21:46:23.678498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678123 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/157ac3ef-1c87-44e5-8c59-edd637bc0c48-installation-pull-secrets\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.678498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678157 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d3596aa9-4bcf-4506-aff0-3706719f73c1-serving-cert\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.678498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678190 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/157ac3ef-1c87-44e5-8c59-edd637bc0c48-image-registry-private-configuration\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.678498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678215 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/157ac3ef-1c87-44e5-8c59-edd637bc0c48-bound-sa-token\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.678498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678282 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/77c6bb84-f632-4621-958c-51b870956328-default-certificate\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.678498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678316 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3596aa9-4bcf-4506-aff0-3706719f73c1-service-ca-bundle\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.678498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678355 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt8jk\" (UniqueName: \"kubernetes.io/projected/d3596aa9-4bcf-4506-aff0-3706719f73c1-kube-api-access-pt8jk\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.678498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678379 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/77c6bb84-f632-4621-958c-51b870956328-stats-auth\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.678498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678404 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52sq7\" (UniqueName: \"kubernetes.io/projected/157ac3ef-1c87-44e5-8c59-edd637bc0c48-kube-api-access-52sq7\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.678498 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678429 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77c6bb84-f632-4621-958c-51b870956328-metrics-certs\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.678976 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678750 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d3596aa9-4bcf-4506-aff0-3706719f73c1-snapshots\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.678976 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678906 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3596aa9-4bcf-4506-aff0-3706719f73c1-service-ca-bundle\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.678976 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678907 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8bc9020b-6a51-4caa-9163-0d474d41b484-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-l9zg5\" (UID: \"8bc9020b-6a51-4caa-9163-0d474d41b484\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" Jul 27 21:46:23.678976 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.678936 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3596aa9-4bcf-4506-aff0-3706719f73c1-tmp\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.679185 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.679010 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3596aa9-4bcf-4506-aff0-3706719f73c1-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.680682 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.680654 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d3596aa9-4bcf-4506-aff0-3706719f73c1-serving-cert\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.680783 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.680654 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bc9020b-6a51-4caa-9163-0d474d41b484-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-l9zg5\" (UID: \"8bc9020b-6a51-4caa-9163-0d474d41b484\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" Jul 27 21:46:23.682442 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.682422 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2"] Jul 27 21:46:23.682544 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.682450 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-774b54654c-9kg49"] Jul 27 21:46:23.682544 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.682519 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" Jul 27 21:46:23.684796 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.684775 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 27 21:46:23.684943 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.684775 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 27 21:46:23.685290 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.685273 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 27 21:46:23.685626 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.685610 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 27 21:46:23.696363 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.696335 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s86p\" (UniqueName: \"kubernetes.io/projected/8bc9020b-6a51-4caa-9163-0d474d41b484-kube-api-access-5s86p\") pod \"cluster-monitoring-operator-548b489cb8-l9zg5\" (UID: \"8bc9020b-6a51-4caa-9163-0d474d41b484\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" Jul 27 21:46:23.697297 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.697276 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt8jk\" (UniqueName: \"kubernetes.io/projected/d3596aa9-4bcf-4506-aff0-3706719f73c1-kube-api-access-pt8jk\") pod \"insights-operator-5885fdb759-b8v9l\" (UID: \"d3596aa9-4bcf-4506-aff0-3706719f73c1\") " pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.704737 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.704717 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:23.779685 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.779607 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/157ac3ef-1c87-44e5-8c59-edd637bc0c48-installation-pull-secrets\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.779685 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.779652 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/157ac3ef-1c87-44e5-8c59-edd637bc0c48-image-registry-private-configuration\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.779685 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.779670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/157ac3ef-1c87-44e5-8c59-edd637bc0c48-bound-sa-token\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.779971 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.779700 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/77c6bb84-f632-4621-958c-51b870956328-default-certificate\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.779971 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.779730 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/77c6bb84-f632-4621-958c-51b870956328-stats-auth\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.779971 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.779749 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52sq7\" (UniqueName: \"kubernetes.io/projected/157ac3ef-1c87-44e5-8c59-edd637bc0c48-kube-api-access-52sq7\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.779971 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.779764 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77c6bb84-f632-4621-958c-51b870956328-metrics-certs\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.779971 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.779781 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/157ac3ef-1c87-44e5-8c59-edd637bc0c48-trusted-ca\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.779971 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.779809 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztgn5\" (UniqueName: \"kubernetes.io/projected/576ee506-01d0-49ec-b07b-196d38eab79c-kube-api-access-ztgn5\") pod \"klusterlet-addon-workmgr-7dbdc9547c-w95h2\" (UID: \"576ee506-01d0-49ec-b07b-196d38eab79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" Jul 27 21:46:23.780298 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.780034 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/157ac3ef-1c87-44e5-8c59-edd637bc0c48-ca-trust-extracted\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.780298 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.780071 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/576ee506-01d0-49ec-b07b-196d38eab79c-tmp\") pod \"klusterlet-addon-workmgr-7dbdc9547c-w95h2\" (UID: \"576ee506-01d0-49ec-b07b-196d38eab79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" Jul 27 21:46:23.780298 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.780122 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77c6bb84-f632-4621-958c-51b870956328-service-ca-bundle\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.780298 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.780153 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/576ee506-01d0-49ec-b07b-196d38eab79c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dbdc9547c-w95h2\" (UID: \"576ee506-01d0-49ec-b07b-196d38eab79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" Jul 27 21:46:23.780298 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.780184 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245s9\" (UniqueName: \"kubernetes.io/projected/77c6bb84-f632-4621-958c-51b870956328-kube-api-access-245s9\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.780298 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.780212 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/157ac3ef-1c87-44e5-8c59-edd637bc0c48-registry-tls\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.780298 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.780235 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/157ac3ef-1c87-44e5-8c59-edd637bc0c48-registry-certificates\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.780665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.780512 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/157ac3ef-1c87-44e5-8c59-edd637bc0c48-ca-trust-extracted\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.781096 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.780954 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77c6bb84-f632-4621-958c-51b870956328-service-ca-bundle\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.781096 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.780954 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/157ac3ef-1c87-44e5-8c59-edd637bc0c48-trusted-ca\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.781290 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.781167 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/157ac3ef-1c87-44e5-8c59-edd637bc0c48-registry-certificates\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.782554 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.782472 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/157ac3ef-1c87-44e5-8c59-edd637bc0c48-image-registry-private-configuration\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.782808 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.782790 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77c6bb84-f632-4621-958c-51b870956328-metrics-certs\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.783012 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.782992 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/77c6bb84-f632-4621-958c-51b870956328-stats-auth\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.783084 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.782991 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/77c6bb84-f632-4621-958c-51b870956328-default-certificate\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.783389 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.783370 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/157ac3ef-1c87-44e5-8c59-edd637bc0c48-installation-pull-secrets\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.783485 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.783471 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/157ac3ef-1c87-44e5-8c59-edd637bc0c48-registry-tls\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.791665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.791647 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/157ac3ef-1c87-44e5-8c59-edd637bc0c48-bound-sa-token\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.791963 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.791944 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52sq7\" (UniqueName: \"kubernetes.io/projected/157ac3ef-1c87-44e5-8c59-edd637bc0c48-kube-api-access-52sq7\") pod \"image-registry-774b54654c-9kg49\" (UID: \"157ac3ef-1c87-44e5-8c59-edd637bc0c48\") " pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.792776 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.792760 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-245s9\" (UniqueName: \"kubernetes.io/projected/77c6bb84-f632-4621-958c-51b870956328-kube-api-access-245s9\") pod \"router-default-5b55db7595-rmmgf\" (UID: \"77c6bb84-f632-4621-958c-51b870956328\") " pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.852773 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.852741 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" Jul 27 21:46:23.863783 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.863753 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:23.867705 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.867691 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-b8v9l" Jul 27 21:46:23.880711 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.880692 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztgn5\" (UniqueName: \"kubernetes.io/projected/576ee506-01d0-49ec-b07b-196d38eab79c-kube-api-access-ztgn5\") pod \"klusterlet-addon-workmgr-7dbdc9547c-w95h2\" (UID: \"576ee506-01d0-49ec-b07b-196d38eab79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" Jul 27 21:46:23.880791 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.880732 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/576ee506-01d0-49ec-b07b-196d38eab79c-tmp\") pod \"klusterlet-addon-workmgr-7dbdc9547c-w95h2\" (UID: \"576ee506-01d0-49ec-b07b-196d38eab79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" Jul 27 21:46:23.880791 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.880757 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/576ee506-01d0-49ec-b07b-196d38eab79c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dbdc9547c-w95h2\" (UID: \"576ee506-01d0-49ec-b07b-196d38eab79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" Jul 27 21:46:23.881125 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.881104 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/576ee506-01d0-49ec-b07b-196d38eab79c-tmp\") pod \"klusterlet-addon-workmgr-7dbdc9547c-w95h2\" (UID: \"576ee506-01d0-49ec-b07b-196d38eab79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" Jul 27 21:46:23.881231 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.881217 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:23.883101 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.883082 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/576ee506-01d0-49ec-b07b-196d38eab79c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dbdc9547c-w95h2\" (UID: \"576ee506-01d0-49ec-b07b-196d38eab79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" Jul 27 21:46:23.888621 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.888602 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztgn5\" (UniqueName: \"kubernetes.io/projected/576ee506-01d0-49ec-b07b-196d38eab79c-kube-api-access-ztgn5\") pod \"klusterlet-addon-workmgr-7dbdc9547c-w95h2\" (UID: \"576ee506-01d0-49ec-b07b-196d38eab79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" Jul 27 21:46:23.957403 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.957334 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:23.970784 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.970686 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:23.996480 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:23.996412 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" Jul 27 21:46:24.021480 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:24.021453 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5"] Jul 27 21:46:24.025718 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:46:24.025678 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc9020b_6a51_4caa_9163_0d474d41b484.slice/crio-c6977d72b71f417bcc484b90b78fbd8337670bf5ec17c44f373c7da1dcc4fb3e WatchSource:0}: Error finding container c6977d72b71f417bcc484b90b78fbd8337670bf5ec17c44f373c7da1dcc4fb3e: Status 404 returned error can't find the container with id c6977d72b71f417bcc484b90b78fbd8337670bf5ec17c44f373c7da1dcc4fb3e Jul 27 21:46:24.042772 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:24.042695 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-b8v9l"] Jul 27 21:46:24.047532 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:46:24.047503 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3596aa9_4bcf_4506_aff0_3706719f73c1.slice/crio-e77d5491f561f26a5abbeef251d9256ac4eecc79c5af8ae34672ee630210c6b3 WatchSource:0}: Error finding container e77d5491f561f26a5abbeef251d9256ac4eecc79c5af8ae34672ee630210c6b3: Status 404 returned error can't find the container with id e77d5491f561f26a5abbeef251d9256ac4eecc79c5af8ae34672ee630210c6b3 Jul 27 21:46:24.070483 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:24.070454 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b55db7595-rmmgf"] Jul 27 21:46:24.074794 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:46:24.074773 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c6bb84_f632_4621_958c_51b870956328.slice/crio-4e983158b78b880ef932c88cbda1458dc17a64e12930b7be8c8bcce25a0f533b WatchSource:0}: Error finding container 4e983158b78b880ef932c88cbda1458dc17a64e12930b7be8c8bcce25a0f533b: Status 404 returned error can't find the container with id 4e983158b78b880ef932c88cbda1458dc17a64e12930b7be8c8bcce25a0f533b Jul 27 21:46:24.088467 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:24.088376 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-b8v9l" event={"ID":"d3596aa9-4bcf-4506-aff0-3706719f73c1","Type":"ContainerStarted","Data":"e77d5491f561f26a5abbeef251d9256ac4eecc79c5af8ae34672ee630210c6b3"} Jul 27 21:46:24.090517 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:24.090491 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" event={"ID":"8bc9020b-6a51-4caa-9163-0d474d41b484","Type":"ContainerStarted","Data":"c6977d72b71f417bcc484b90b78fbd8337670bf5ec17c44f373c7da1dcc4fb3e"} Jul 27 21:46:24.092402 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:24.092375 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b55db7595-rmmgf" event={"ID":"77c6bb84-f632-4621-958c-51b870956328","Type":"ContainerStarted","Data":"4e983158b78b880ef932c88cbda1458dc17a64e12930b7be8c8bcce25a0f533b"} Jul 27 21:46:24.144090 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:24.144063 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-774b54654c-9kg49"] Jul 27 21:46:24.146988 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:46:24.146965 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157ac3ef_1c87_44e5_8c59_edd637bc0c48.slice/crio-f8f6acc521d63aa7952f8e0fb86d37a6cf0981ebe2960b27c0e681e8a5f09628 WatchSource:0}: Error finding container f8f6acc521d63aa7952f8e0fb86d37a6cf0981ebe2960b27c0e681e8a5f09628: Status 404 returned error can't find the container with id f8f6acc521d63aa7952f8e0fb86d37a6cf0981ebe2960b27c0e681e8a5f09628 Jul 27 21:46:24.160924 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:24.160895 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2"] Jul 27 21:46:24.164677 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:46:24.164647 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576ee506_01d0_49ec_b07b_196d38eab79c.slice/crio-948327fe73d2cee91d940c7a5cd43b809b0ab60af3e54b679dd95551e83c57fa WatchSource:0}: Error finding container 948327fe73d2cee91d940c7a5cd43b809b0ab60af3e54b679dd95551e83c57fa: Status 404 returned error can't find the container with id 948327fe73d2cee91d940c7a5cd43b809b0ab60af3e54b679dd95551e83c57fa Jul 27 21:46:24.441421 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:24.441389 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:24.513654 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:24.513628 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4kt4" Jul 27 21:46:24.523383 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:24.523361 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:24.624170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:24.624135 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:24.943717 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:24.943682 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:25.044019 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:25.043987 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:25.100670 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:25.100628 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" event={"ID":"576ee506-01d0-49ec-b07b-196d38eab79c","Type":"ContainerStarted","Data":"948327fe73d2cee91d940c7a5cd43b809b0ab60af3e54b679dd95551e83c57fa"} Jul 27 21:46:25.102370 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:25.102343 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-774b54654c-9kg49" event={"ID":"157ac3ef-1c87-44e5-8c59-edd637bc0c48","Type":"ContainerStarted","Data":"c75f8bca3d8eee2fb838f1a3023bb812a70d2d49a869b4c733ec7d2101e9417e"} Jul 27 21:46:25.102509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:25.102390 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-774b54654c-9kg49" event={"ID":"157ac3ef-1c87-44e5-8c59-edd637bc0c48","Type":"ContainerStarted","Data":"f8f6acc521d63aa7952f8e0fb86d37a6cf0981ebe2960b27c0e681e8a5f09628"} Jul 27 21:46:25.102509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:25.102459 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:25.104014 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:25.103986 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b55db7595-rmmgf" event={"ID":"77c6bb84-f632-4621-958c-51b870956328","Type":"ContainerStarted","Data":"170468412c0b52227a47a037308f5f059f26d1758f33cfb614be4d4606670283"} Jul 27 21:46:25.121396 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:25.121342 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-774b54654c-9kg49" podStartSLOduration=2.121322585 podStartE2EDuration="2.121322585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:25.12069871 +0000 UTC m=+62.738136033" watchObservedRunningTime="2026-07-27 21:46:25.121322585 +0000 UTC m=+62.738759904" Jul 27 21:46:25.138489 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:25.138439 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5b55db7595-rmmgf" podStartSLOduration=2.138418972 podStartE2EDuration="2.138418972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:25.137650226 +0000 UTC m=+62.755087546" watchObservedRunningTime="2026-07-27 21:46:25.138418972 +0000 UTC m=+62.755856285" Jul 27 21:46:25.748665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:25.748594 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:25.848842 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:25.848809 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:25.882234 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:25.882203 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:25.885092 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:25.885043 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:26.067232 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:26.067155 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c77xc" Jul 27 21:46:26.118134 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:26.117669 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:26.119665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:26.119640 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5b55db7595-rmmgf" Jul 27 21:46:26.180503 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:26.180474 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:26.273864 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:26.273825 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:28.529104 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.529059 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs\") pod \"network-metrics-daemon-p79rn\" (UID: \"bf30d435-7854-4c24-b72a-c6992b0776a7\") " pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:46:28.531470 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.531445 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 21:46:28.542502 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.542480 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf30d435-7854-4c24-b72a-c6992b0776a7-metrics-certs\") pod \"network-metrics-daemon-p79rn\" (UID: \"bf30d435-7854-4c24-b72a-c6992b0776a7\") " pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:46:28.630562 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.630522 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:46:28.633074 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.633047 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 21:46:28.643278 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.643240 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ef19975-ecaa-49ba-9cd9-e16737e46c47-original-pull-secret\") pod \"global-pull-secret-syncer-vlh9m\" (UID: \"1ef19975-ecaa-49ba-9cd9-e16737e46c47\") " pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:46:28.726490 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.726462 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cj7gv\"" Jul 27 21:46:28.731059 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.730917 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692z2\" (UniqueName: \"kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2\") pod \"network-check-target-w86pt\" (UID: \"5bdac9f5-97c0-4a7f-9ec7-dd07f0343253\") " pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:46:28.732989 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.732973 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 21:46:28.734358 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.734343 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p79rn" Jul 27 21:46:28.743572 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.743551 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 21:46:28.754819 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.754797 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-692z2\" (UniqueName: \"kubernetes.io/projected/5bdac9f5-97c0-4a7f-9ec7-dd07f0343253-kube-api-access-692z2\") pod \"network-check-target-w86pt\" (UID: \"5bdac9f5-97c0-4a7f-9ec7-dd07f0343253\") " pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:46:28.824134 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.824099 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlh9m" Jul 27 21:46:28.946284 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:28.945604 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p79rn"] Jul 27 21:46:29.010713 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.010683 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vlh9m"] Jul 27 21:46:29.016287 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:46:29.016223 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef19975_ecaa_49ba_9cd9_e16737e46c47.slice/crio-defe4ca609a66221e9daedd98725f177a066d59122e2ff4058552ff60a8607a0 WatchSource:0}: Error finding container defe4ca609a66221e9daedd98725f177a066d59122e2ff4058552ff60a8607a0: Status 404 returned error can't find the container with id defe4ca609a66221e9daedd98725f177a066d59122e2ff4058552ff60a8607a0 Jul 27 21:46:29.032402 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.032351 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kt796\"" Jul 27 21:46:29.040764 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.040733 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:46:29.127552 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.126867 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-b8v9l" event={"ID":"d3596aa9-4bcf-4506-aff0-3706719f73c1","Type":"ContainerStarted","Data":"a3026c03225570d0b8a107204930b07b347f5b42bbfd762365f7df1eff40e755"} Jul 27 21:46:29.131736 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.131666 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" event={"ID":"8bc9020b-6a51-4caa-9163-0d474d41b484","Type":"ContainerStarted","Data":"624e7b120f6f27c474c7394e6061c305fa3f6aa7621a41dfae06b88f5a3669e6"} Jul 27 21:46:29.133415 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.133150 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vlh9m" event={"ID":"1ef19975-ecaa-49ba-9cd9-e16737e46c47","Type":"ContainerStarted","Data":"defe4ca609a66221e9daedd98725f177a066d59122e2ff4058552ff60a8607a0"} Jul 27 21:46:29.134945 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.134673 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" event={"ID":"576ee506-01d0-49ec-b07b-196d38eab79c","Type":"ContainerStarted","Data":"148fb932562e73b1908058d651623ac0be04d7af75a8ea7a2d7179198ad255af"} Jul 27 21:46:29.135370 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.135320 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" Jul 27 21:46:29.138171 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.136231 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p79rn" event={"ID":"bf30d435-7854-4c24-b72a-c6992b0776a7","Type":"ContainerStarted","Data":"d76ba26b606c42ae978d12ecd597074bfd46255cb92c4d9518765bb3ca24a1d8"} Jul 27 21:46:29.138171 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.137308 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:29.140212 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.140177 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" Jul 27 21:46:29.146501 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.145870 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-b8v9l" podStartSLOduration=1.459156597 podStartE2EDuration="6.145854889s" totalImagesPullingTime="4.686698292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:29.145491346 +0000 UTC m=+66.762928667" watchObservedRunningTime="2026-07-27 21:46:29.145854889 +0000 UTC m=+66.763292209" Jul 27 21:46:29.168543 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.168484 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-l9zg5" podStartSLOduration=1.466159343 podStartE2EDuration="6.168467213s" totalImagesPullingTime="4.70230787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:29.167810469 +0000 UTC m=+66.785247790" watchObservedRunningTime="2026-07-27 21:46:29.168467213 +0000 UTC m=+66.785904535" Jul 27 21:46:29.181224 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.181195 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w86pt"] Jul 27 21:46:29.185112 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:46:29.185073 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdac9f5_97c0_4a7f_9ec7_dd07f0343253.slice/crio-961ee9bc6e295ba462aa7445a542835e2c94876b8ee49cec5e7880516e658f4f WatchSource:0}: Error finding container 961ee9bc6e295ba462aa7445a542835e2c94876b8ee49cec5e7880516e658f4f: Status 404 returned error can't find the container with id 961ee9bc6e295ba462aa7445a542835e2c94876b8ee49cec5e7880516e658f4f Jul 27 21:46:29.235308 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.235192 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:29.328581 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.328473 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:29.557276 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.556281 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbdc9547c-w95h2" podStartSLOduration=1.936735101 podStartE2EDuration="6.556241053s" totalImagesPullingTime="4.619505952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:29.183708079 +0000 UTC m=+66.801145399" watchObservedRunningTime="2026-07-27 21:46:29.556241053 +0000 UTC m=+67.173678377" Jul 27 21:46:29.557746 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.557679 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w78fm"] Jul 27 21:46:29.566469 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.565614 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.569332 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.569187 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7smr8\"" Jul 27 21:46:29.569930 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.569865 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 21:46:29.570449 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.570311 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 21:46:29.577229 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.577155 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w78fm"] Jul 27 21:46:29.597856 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.597380 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:29.638119 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.638077 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a416c09a-03d4-4e4f-995d-217588936dde-data-volume\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.638312 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.638128 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n48fv\" (UniqueName: \"kubernetes.io/projected/a416c09a-03d4-4e4f-995d-217588936dde-kube-api-access-n48fv\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.638312 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.638157 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.638312 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.638183 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.638312 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.638219 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a416c09a-03d4-4e4f-995d-217588936dde-crio-socket\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.656328 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.656238 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:29.740585 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.739142 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.740585 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.739200 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.740585 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.739245 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a416c09a-03d4-4e4f-995d-217588936dde-crio-socket\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.740585 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.739352 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a416c09a-03d4-4e4f-995d-217588936dde-data-volume\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.740585 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.739380 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n48fv\" (UniqueName: \"kubernetes.io/projected/a416c09a-03d4-4e4f-995d-217588936dde-kube-api-access-n48fv\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.740585 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:29.739805 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:46:29.740585 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:29.739880 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm podName:a416c09a-03d4-4e4f-995d-217588936dde nodeName:}" failed. No retries permitted until 2026-07-27 21:46:30.239856615 +0000 UTC m=+67.857293927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w78fm" (UID: "a416c09a-03d4-4e4f-995d-217588936dde") : configmap "kube-rbac-proxy" not found Jul 27 21:46:29.740585 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:29.740125 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:46:29.740585 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:29.740167 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls podName:a416c09a-03d4-4e4f-995d-217588936dde nodeName:}" failed. No retries permitted until 2026-07-27 21:46:30.24015388 +0000 UTC m=+67.857591183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w78fm" (UID: "a416c09a-03d4-4e4f-995d-217588936dde") : secret "insights-runtime-extractor-tls" not found Jul 27 21:46:29.740585 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.740232 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a416c09a-03d4-4e4f-995d-217588936dde-crio-socket\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.740585 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.740585 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a416c09a-03d4-4e4f-995d-217588936dde-data-volume\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.752940 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.752910 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n48fv\" (UniqueName: \"kubernetes.io/projected/a416c09a-03d4-4e4f-995d-217588936dde-kube-api-access-n48fv\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:29.890740 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.890616 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:29.994148 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:29.994111 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:30.066107 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:30.066076 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:30.141748 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:30.141686 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w86pt" event={"ID":"5bdac9f5-97c0-4a7f-9ec7-dd07f0343253","Type":"ContainerStarted","Data":"961ee9bc6e295ba462aa7445a542835e2c94876b8ee49cec5e7880516e658f4f"} Jul 27 21:46:30.242847 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:30.242816 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:30.244292 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:30.242874 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:30.244292 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:30.243409 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:46:30.244292 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:30.243464 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm podName:a416c09a-03d4-4e4f-995d-217588936dde nodeName:}" failed. No retries permitted until 2026-07-27 21:46:31.243445191 +0000 UTC m=+68.860882493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w78fm" (UID: "a416c09a-03d4-4e4f-995d-217588936dde") : configmap "kube-rbac-proxy" not found Jul 27 21:46:30.244292 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:30.243854 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:46:30.244292 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:30.243908 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls podName:a416c09a-03d4-4e4f-995d-217588936dde nodeName:}" failed. No retries permitted until 2026-07-27 21:46:31.243890109 +0000 UTC m=+68.861327406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w78fm" (UID: "a416c09a-03d4-4e4f-995d-217588936dde") : secret "insights-runtime-extractor-tls" not found Jul 27 21:46:30.570637 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:30.570607 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c77xc_3c159765-958d-4936-a11a-092a2566089b/dns/0.log" Jul 27 21:46:30.770274 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:30.770234 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c77xc_3c159765-958d-4936-a11a-092a2566089b/kube-rbac-proxy/0.log" Jul 27 21:46:31.147564 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:31.147523 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p79rn" event={"ID":"bf30d435-7854-4c24-b72a-c6992b0776a7","Type":"ContainerStarted","Data":"bb706bf48969f6576eaa5b3b07c581476a6f26fb141956cf80cc3e2f7efdb3f8"} Jul 27 21:46:31.147741 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:31.147586 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p79rn" event={"ID":"bf30d435-7854-4c24-b72a-c6992b0776a7","Type":"ContainerStarted","Data":"b6bc7cea55b2b7df887877aa22914ad92386cf314dd75fff0f96a7d75ff51d2c"} Jul 27 21:46:31.164075 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:31.164023 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p79rn" podStartSLOduration=67.986590226 podStartE2EDuration="1m9.164007843s" totalImagesPullingTime="1.177417617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:31.162497922 +0000 UTC m=+68.779935242" watchObservedRunningTime="2026-07-27 21:46:31.164007843 +0000 UTC m=+68.781445162" Jul 27 21:46:31.252624 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:31.252560 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:31.252624 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:31.252619 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:31.252866 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:31.252702 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:46:31.252866 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:31.252769 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm podName:a416c09a-03d4-4e4f-995d-217588936dde nodeName:}" failed. No retries permitted until 2026-07-27 21:46:33.252751604 +0000 UTC m=+70.870188902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w78fm" (UID: "a416c09a-03d4-4e4f-995d-217588936dde") : configmap "kube-rbac-proxy" not found Jul 27 21:46:31.253030 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:31.252952 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:46:31.253076 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:31.253038 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls podName:a416c09a-03d4-4e4f-995d-217588936dde nodeName:}" failed. No retries permitted until 2026-07-27 21:46:33.253019558 +0000 UTC m=+70.870456858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w78fm" (UID: "a416c09a-03d4-4e4f-995d-217588936dde") : secret "insights-runtime-extractor-tls" not found Jul 27 21:46:31.282577 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:31.282537 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:31.369693 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:31.369668 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8m8z2_7f949673-96be-4e7f-9cf8-5f69124a6309/dns-node-resolver/0.log" Jul 27 21:46:31.381428 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:31.381395 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:32.170392 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:32.170361 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-774b54654c-9kg49_157ac3ef-1c87-44e5-8c59-edd637bc0c48/registry/0.log" Jul 27 21:46:32.370641 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:32.370595 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-986d7665c-s5xvq_9cd55b06-4d11-468a-9991-473beba0001b/registry/0.log" Jul 27 21:46:32.594770 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:32.594688 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:32.658127 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:32.658093 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:32.770553 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:32.770529 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ghrhz_8aa2c150-1b16-43c8-81a7-a290299036d3/node-ca/0.log" Jul 27 21:46:33.110130 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:33.110088 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:33.174244 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:33.174186 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b55db7595-rmmgf_77c6bb84-f632-4621-958c-51b870956328/router/0.log" Jul 27 21:46:33.218191 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:33.218160 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:33.271993 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:33.271950 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:33.271993 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:33.271996 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:33.272243 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:33.272086 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:46:33.272243 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:33.272153 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm podName:a416c09a-03d4-4e4f-995d-217588936dde nodeName:}" failed. No retries permitted until 2026-07-27 21:46:37.272134571 +0000 UTC m=+74.889571884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w78fm" (UID: "a416c09a-03d4-4e4f-995d-217588936dde") : configmap "kube-rbac-proxy" not found Jul 27 21:46:33.272243 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:33.272154 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:46:33.272243 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:33.272189 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls podName:a416c09a-03d4-4e4f-995d-217588936dde nodeName:}" failed. No retries permitted until 2026-07-27 21:46:37.272180755 +0000 UTC m=+74.889618056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w78fm" (UID: "a416c09a-03d4-4e4f-995d-217588936dde") : secret "insights-runtime-extractor-tls" not found Jul 27 21:46:33.388105 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:33.388026 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:33.457337 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:33.457303 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:33.515203 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:33.515166 2561 patch_prober.go:28] interesting pod/image-registry-986d7665c-s5xvq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 27 21:46:33.515376 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:33.515230 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-986d7665c-s5xvq" podUID="9cd55b06-4d11-468a-9991-473beba0001b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:46:33.770008 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:33.769956 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z2qwk_8529dd01-316a-4c4b-af14-e04511a99734/serve-healthcheck-canary/0.log" Jul 27 21:46:33.897094 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:33.897061 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:34.160248 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:34.160215 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vlh9m" event={"ID":"1ef19975-ecaa-49ba-9cd9-e16737e46c47","Type":"ContainerStarted","Data":"c6d63479044e9e42b8ee5b6271fe2a34db1744d940265157bf6da91760be5ef7"} Jul 27 21:46:34.161663 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:34.161636 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w86pt" event={"ID":"5bdac9f5-97c0-4a7f-9ec7-dd07f0343253","Type":"ContainerStarted","Data":"e949f0e210537b0c5106e1d76903c512f7e3c7d691f02317d342fe9b9f4aae79"} Jul 27 21:46:34.161782 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:34.161760 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:46:34.174804 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:34.174763 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vlh9m" podStartSLOduration=66.278717208 podStartE2EDuration="1m11.174749806s" totalImagesPullingTime="4.896032598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:34.173925729 +0000 UTC m=+71.791363049" watchObservedRunningTime="2026-07-27 21:46:34.174749806 +0000 UTC m=+71.792187124" Jul 27 21:46:34.188453 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:34.188402 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w86pt" podStartSLOduration=67.465794503 podStartE2EDuration="1m12.188385418s" totalImagesPullingTime="4.722590915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:34.187357509 +0000 UTC m=+71.804794839" watchObservedRunningTime="2026-07-27 21:46:34.188385418 +0000 UTC m=+71.805822738" Jul 27 21:46:34.351573 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:34.351540 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:34.451210 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:34.451126 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:35.203172 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:35.203136 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:35.283572 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:35.283541 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:36.119769 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:36.119739 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:36.298777 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:36.298747 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:36.363995 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:36.363966 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:36.473166 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:36.473135 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:36.533495 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:36.533461 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:37.181936 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.181907 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:37.234690 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.234664 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:37.306041 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.306008 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:37.306402 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.306053 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:37.306402 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:37.306144 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:46:37.306402 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:37.306177 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:46:37.306402 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:37.306214 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm podName:a416c09a-03d4-4e4f-995d-217588936dde nodeName:}" failed. No retries permitted until 2026-07-27 21:46:45.306197406 +0000 UTC m=+82.923634717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w78fm" (UID: "a416c09a-03d4-4e4f-995d-217588936dde") : configmap "kube-rbac-proxy" not found Jul 27 21:46:37.306402 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:37.306233 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls podName:a416c09a-03d4-4e4f-995d-217588936dde nodeName:}" failed. No retries permitted until 2026-07-27 21:46:45.30622716 +0000 UTC m=+82.923664456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w78fm" (UID: "a416c09a-03d4-4e4f-995d-217588936dde") : secret "insights-runtime-extractor-tls" not found Jul 27 21:46:37.502048 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.501968 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:37.723592 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.723550 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:37.749529 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.749493 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pqblm"] Jul 27 21:46:37.752486 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.752424 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.754999 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.754973 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 21:46:37.755298 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.755280 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xtsgm\"" Jul 27 21:46:37.755406 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.755384 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 21:46:37.755564 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.755539 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 21:46:37.755667 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.755615 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 21:46:37.811585 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.811553 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07edee64-5950-439c-9488-6c0c7a5f0191-metrics-client-ca\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.811769 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.811591 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-accelerators-collector-config\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.811769 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.811643 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-wtmp\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.811769 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.811693 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-textfile\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.811769 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.811733 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-tls\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.811977 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.811790 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07edee64-5950-439c-9488-6c0c7a5f0191-sys\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.811977 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.811833 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlb2v\" (UniqueName: \"kubernetes.io/projected/07edee64-5950-439c-9488-6c0c7a5f0191-kube-api-access-qlb2v\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.811977 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.811862 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07edee64-5950-439c-9488-6c0c7a5f0191-root\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.811977 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.811899 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.819771 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.819727 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:37.913091 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913046 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.913091 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913101 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07edee64-5950-439c-9488-6c0c7a5f0191-metrics-client-ca\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.913091 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-accelerators-collector-config\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.913451 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913145 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-wtmp\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.913451 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913175 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-textfile\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.913451 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913216 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-tls\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.913451 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913279 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07edee64-5950-439c-9488-6c0c7a5f0191-sys\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.913451 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913316 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-wtmp\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.913451 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913317 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlb2v\" (UniqueName: \"kubernetes.io/projected/07edee64-5950-439c-9488-6c0c7a5f0191-kube-api-access-qlb2v\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.913451 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913376 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07edee64-5950-439c-9488-6c0c7a5f0191-root\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.913451 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913450 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07edee64-5950-439c-9488-6c0c7a5f0191-root\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.913879 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:37.913528 2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 27 21:46:37.913879 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:46:37.913577 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-tls podName:07edee64-5950-439c-9488-6c0c7a5f0191 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:38.413560755 +0000 UTC m=+76.030998063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-tls") pod "node-exporter-pqblm" (UID: "07edee64-5950-439c-9488-6c0c7a5f0191") : secret "node-exporter-tls" not found Jul 27 21:46:37.913879 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913623 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-textfile\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.913879 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913645 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07edee64-5950-439c-9488-6c0c7a5f0191-sys\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.913879 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.913838 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-accelerators-collector-config\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.914116 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.914067 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07edee64-5950-439c-9488-6c0c7a5f0191-metrics-client-ca\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.915947 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.915920 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:37.921866 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.921843 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:37.923617 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:37.923599 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlb2v\" (UniqueName: \"kubernetes.io/projected/07edee64-5950-439c-9488-6c0c7a5f0191-kube-api-access-qlb2v\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:38.019946 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:38.019844 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:38.121191 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:38.121161 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:38.198956 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:38.198922 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:38.255170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:38.255139 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:38.355781 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:38.355690 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:38.418128 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:38.418092 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-tls\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:38.420360 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:38.420343 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07edee64-5950-439c-9488-6c0c7a5f0191-node-exporter-tls\") pod \"node-exporter-pqblm\" (UID: \"07edee64-5950-439c-9488-6c0c7a5f0191\") " pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:38.493991 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:38.493947 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:38.663489 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:38.663454 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pqblm" Jul 27 21:46:38.673628 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:46:38.673604 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07edee64_5950_439c_9488_6c0c7a5f0191.slice/crio-70b323b79d43fc89cec1cbfdc0270d97fbfb7c1021b3d1b02a700bd67ddc992c WatchSource:0}: Error finding container 70b323b79d43fc89cec1cbfdc0270d97fbfb7c1021b3d1b02a700bd67ddc992c: Status 404 returned error can't find the container with id 70b323b79d43fc89cec1cbfdc0270d97fbfb7c1021b3d1b02a700bd67ddc992c Jul 27 21:46:38.742561 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:38.742527 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:38.835104 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:38.835068 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:38.980643 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:38.980558 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:39.180900 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:39.180868 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pqblm" event={"ID":"07edee64-5950-439c-9488-6c0c7a5f0191","Type":"ContainerStarted","Data":"70b323b79d43fc89cec1cbfdc0270d97fbfb7c1021b3d1b02a700bd67ddc992c"} Jul 27 21:46:40.187113 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:40.187078 2561 generic.go:356] "Generic (PLEG): container finished" podID="07edee64-5950-439c-9488-6c0c7a5f0191" containerID="6af2887b7bf8cb3ac09051311f743b3eaaacc8b36658137710cee1f5fc8ace8a" exitCode=0 Jul 27 21:46:40.187515 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:40.187123 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pqblm" event={"ID":"07edee64-5950-439c-9488-6c0c7a5f0191","Type":"ContainerDied","Data":"6af2887b7bf8cb3ac09051311f743b3eaaacc8b36658137710cee1f5fc8ace8a"} Jul 27 21:46:40.739474 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:40.739443 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:40.798313 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:40.798282 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:40.902873 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:40.902844 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:41.000324 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:41.000245 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:41.192603 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:41.192567 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pqblm" event={"ID":"07edee64-5950-439c-9488-6c0c7a5f0191","Type":"ContainerStarted","Data":"453ca42aa1b0c6945864fa7e99e59fb7dc6ee004d0e5abc4bb841ba8c4465082"} Jul 27 21:46:41.192991 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:41.192629 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pqblm" event={"ID":"07edee64-5950-439c-9488-6c0c7a5f0191","Type":"ContainerStarted","Data":"eff4ea31eb1a56b14fc79952b749b23e7ad14f4d1713fd2c4b5f7b4bb9e18e48"} Jul 27 21:46:41.212865 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:41.212816 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pqblm" podStartSLOduration=3.082685892 podStartE2EDuration="4.212801554s" totalImagesPullingTime="1.061838275s" totalInitContainerRuntime="68.277387ms" isStatefulPod=true podCreationTimestamp="2026-07-27 21:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:41.211357494 +0000 UTC m=+78.828794813" watchObservedRunningTime="2026-07-27 21:46:41.212801554 +0000 UTC m=+78.830238873" Jul 27 21:46:41.777105 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:41.777072 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:42.538966 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:42.538938 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:42.620964 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:42.620929 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:42.750553 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:42.750524 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:43.470506 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:43.470467 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:43.512391 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:43.512365 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:46:43.554201 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:43.554173 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:43.623622 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:43.623596 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:44.102850 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:44.102816 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:45.381514 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:45.381392 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:45.381514 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:45.381432 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:45.381933 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:45.381916 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a416c09a-03d4-4e4f-995d-217588936dde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:45.383664 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:45.383635 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a416c09a-03d4-4e4f-995d-217588936dde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w78fm\" (UID: \"a416c09a-03d4-4e4f-995d-217588936dde\") " pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:45.490865 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:45.490831 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w78fm" Jul 27 21:46:45.543166 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:45.543133 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:45.617010 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:45.616978 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w78fm"] Jul 27 21:46:45.619976 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:46:45.619932 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda416c09a_03d4_4e4f_995d_217588936dde.slice/crio-9ac7e22ae66bf592b9d0f6e6ed93a31058ed8d91cb1047e938eae5087e5f63f8 WatchSource:0}: Error finding container 9ac7e22ae66bf592b9d0f6e6ed93a31058ed8d91cb1047e938eae5087e5f63f8: Status 404 returned error can't find the container with id 9ac7e22ae66bf592b9d0f6e6ed93a31058ed8d91cb1047e938eae5087e5f63f8 Jul 27 21:46:45.620805 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:45.620460 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:46.123562 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:46.122738 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-774b54654c-9kg49" Jul 27 21:46:46.217139 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:46.217104 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w78fm" event={"ID":"a416c09a-03d4-4e4f-995d-217588936dde","Type":"ContainerStarted","Data":"9ac7e22ae66bf592b9d0f6e6ed93a31058ed8d91cb1047e938eae5087e5f63f8"} Jul 27 21:46:46.271250 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:46.271173 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:46.371495 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:46.371466 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:47.345427 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:47.345388 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:47.528053 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:47.528024 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:48.112324 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:48.112295 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:48.168693 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:48.168663 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:48.525756 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:48.525716 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-986d7665c-s5xvq" podUID="9cd55b06-4d11-468a-9991-473beba0001b" containerName="registry" containerID="cri-o://89560eb49f972945beb9970410f1a1d8007d9a7b659e6dab4d5472eb5642b6fc" gracePeriod=30 Jul 27 21:46:49.386229 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:49.386200 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:49.450480 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:49.450450 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:50.234443 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.234415 2561 generic.go:356] "Generic (PLEG): container finished" podID="9cd55b06-4d11-468a-9991-473beba0001b" containerID="89560eb49f972945beb9970410f1a1d8007d9a7b659e6dab4d5472eb5642b6fc" exitCode=0 Jul 27 21:46:50.234848 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.234502 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-986d7665c-s5xvq" event={"ID":"9cd55b06-4d11-468a-9991-473beba0001b","Type":"ContainerDied","Data":"89560eb49f972945beb9970410f1a1d8007d9a7b659e6dab4d5472eb5642b6fc"} Jul 27 21:46:50.344717 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.344695 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:46:50.427064 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.427030 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-bound-sa-token\") pod \"9cd55b06-4d11-468a-9991-473beba0001b\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " Jul 27 21:46:50.427200 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.427085 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cd55b06-4d11-468a-9991-473beba0001b-installation-pull-secrets\") pod \"9cd55b06-4d11-468a-9991-473beba0001b\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " Jul 27 21:46:50.427200 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.427141 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls\") pod \"9cd55b06-4d11-468a-9991-473beba0001b\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " Jul 27 21:46:50.427200 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.427165 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cd55b06-4d11-468a-9991-473beba0001b-ca-trust-extracted\") pod \"9cd55b06-4d11-468a-9991-473beba0001b\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " Jul 27 21:46:50.427200 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.427190 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cd55b06-4d11-468a-9991-473beba0001b-registry-certificates\") pod \"9cd55b06-4d11-468a-9991-473beba0001b\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " Jul 27 21:46:50.427447 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.427219 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pdbd\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-kube-api-access-9pdbd\") pod \"9cd55b06-4d11-468a-9991-473beba0001b\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " Jul 27 21:46:50.427447 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.427247 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cd55b06-4d11-468a-9991-473beba0001b-image-registry-private-configuration\") pod \"9cd55b06-4d11-468a-9991-473beba0001b\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " Jul 27 21:46:50.427447 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.427308 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cd55b06-4d11-468a-9991-473beba0001b-trusted-ca\") pod \"9cd55b06-4d11-468a-9991-473beba0001b\" (UID: \"9cd55b06-4d11-468a-9991-473beba0001b\") " Jul 27 21:46:50.428126 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.428096 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd55b06-4d11-468a-9991-473beba0001b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9cd55b06-4d11-468a-9991-473beba0001b" (UID: "9cd55b06-4d11-468a-9991-473beba0001b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:46:50.428229 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.428132 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd55b06-4d11-468a-9991-473beba0001b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9cd55b06-4d11-468a-9991-473beba0001b" (UID: "9cd55b06-4d11-468a-9991-473beba0001b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:46:50.430140 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.430099 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-kube-api-access-9pdbd" (OuterVolumeSpecName: "kube-api-access-9pdbd") pod "9cd55b06-4d11-468a-9991-473beba0001b" (UID: "9cd55b06-4d11-468a-9991-473beba0001b"). InnerVolumeSpecName "kube-api-access-9pdbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:46:50.430249 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.430198 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9cd55b06-4d11-468a-9991-473beba0001b" (UID: "9cd55b06-4d11-468a-9991-473beba0001b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:46:50.430326 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.430291 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9cd55b06-4d11-468a-9991-473beba0001b" (UID: "9cd55b06-4d11-468a-9991-473beba0001b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:46:50.430489 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.430464 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd55b06-4d11-468a-9991-473beba0001b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9cd55b06-4d11-468a-9991-473beba0001b" (UID: "9cd55b06-4d11-468a-9991-473beba0001b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:46:50.430558 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.430539 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd55b06-4d11-468a-9991-473beba0001b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9cd55b06-4d11-468a-9991-473beba0001b" (UID: "9cd55b06-4d11-468a-9991-473beba0001b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:46:50.436546 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.436515 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd55b06-4d11-468a-9991-473beba0001b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9cd55b06-4d11-468a-9991-473beba0001b" (UID: "9cd55b06-4d11-468a-9991-473beba0001b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:46:50.528665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.528573 2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-registry-tls\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:46:50.528665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.528606 2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cd55b06-4d11-468a-9991-473beba0001b-ca-trust-extracted\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:46:50.528665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.528616 2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cd55b06-4d11-468a-9991-473beba0001b-registry-certificates\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:46:50.528665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.528627 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pdbd\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-kube-api-access-9pdbd\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:46:50.528665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.528642 2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cd55b06-4d11-468a-9991-473beba0001b-image-registry-private-configuration\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:46:50.528665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.528659 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cd55b06-4d11-468a-9991-473beba0001b-trusted-ca\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:46:50.528665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.528666 2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cd55b06-4d11-468a-9991-473beba0001b-bound-sa-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:46:50.528665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:50.528674 2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cd55b06-4d11-468a-9991-473beba0001b-installation-pull-secrets\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:46:51.239910 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:51.239877 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-986d7665c-s5xvq" Jul 27 21:46:51.240347 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:51.239877 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-986d7665c-s5xvq" event={"ID":"9cd55b06-4d11-468a-9991-473beba0001b","Type":"ContainerDied","Data":"5c714e281091d60b81d8a7a5e8832c6931b1ce51a950a30c3c3f0530251cd8af"} Jul 27 21:46:51.240347 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:51.240029 2561 scope.go:117] "RemoveContainer" containerID="89560eb49f972945beb9970410f1a1d8007d9a7b659e6dab4d5472eb5642b6fc" Jul 27 21:46:51.242674 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:51.242634 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w78fm" event={"ID":"a416c09a-03d4-4e4f-995d-217588936dde","Type":"ContainerStarted","Data":"ae66c0cc94e272a8eb7a7e662d9b9c0bda7582721f92efa4bca43f46d4cf3780"} Jul 27 21:46:51.256798 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:51.256776 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-986d7665c-s5xvq"] Jul 27 21:46:51.260213 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:51.260191 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-986d7665c-s5xvq"] Jul 27 21:46:52.818551 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:52.818520 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd55b06-4d11-468a-9991-473beba0001b" path="/var/lib/kubelet/pods/9cd55b06-4d11-468a-9991-473beba0001b/volumes" Jul 27 21:46:53.788146 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:53.788110 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:54.257579 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:54.257538 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w78fm" event={"ID":"a416c09a-03d4-4e4f-995d-217588936dde","Type":"ContainerStarted","Data":"a6c1d99d9374765943fd1af51764a6ad4ff77452cab6e9ba459a2163ee7c7fe9"} Jul 27 21:46:56.124980 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:56.124950 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:56.225946 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:56.225889 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:56.398523 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:56.398492 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:56.452761 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:56.452725 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:46:57.270230 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:57.270198 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w78fm" event={"ID":"a416c09a-03d4-4e4f-995d-217588936dde","Type":"ContainerStarted","Data":"9e3aa4da68b00b1a3814d4dec30ca83499f48e1f973e6e8946cbc02c1116965b"} Jul 27 21:46:57.288401 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:46:57.288349 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w78fm" podStartSLOduration=16.976300828 podStartE2EDuration="28.288332304s" totalImagesPullingTime="11.312031476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:46:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:57.288013247 +0000 UTC m=+94.905450565" watchObservedRunningTime="2026-07-27 21:46:57.288332304 +0000 UTC m=+94.905769623" Jul 27 21:47:02.674509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:02.674475 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:02.775488 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:02.775452 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:02.825687 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:02.825646 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:03.122441 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:03.122359 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:03.196159 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:03.196114 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:05.166838 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:05.166804 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w86pt" Jul 27 21:47:18.113596 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:18.113562 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:18.174074 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:18.174031 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:19.319655 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:19.319618 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:19.393084 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:19.393056 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:25.094313 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:25.094253 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:25.196212 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:25.196173 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:25.300461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:25.300429 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:53.927253 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:53.927221 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:54.013564 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:54.013531 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:58.157877 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:58.157843 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:58.260981 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:58.260950 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:59.803442 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:59.803413 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:59.891775 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:47:59.891745 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:05.865256 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:05.865220 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:05.918223 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:05.918193 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:08.970162 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:08.970097 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:09.072546 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:09.072512 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:09.140021 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:09.139994 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:18.163621 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:18.163544 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:18.264608 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:18.264583 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:38.504542 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:38.504509 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:38.645302 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:38.645253 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:40.109886 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.109849 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc"] Jul 27 21:48:40.110255 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.110093 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd55b06-4d11-468a-9991-473beba0001b" containerName="registry" Jul 27 21:48:40.110255 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.110103 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd55b06-4d11-468a-9991-473beba0001b" containerName="registry" Jul 27 21:48:40.110255 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.110151 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd55b06-4d11-468a-9991-473beba0001b" containerName="registry" Jul 27 21:48:40.113113 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.113097 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" Jul 27 21:48:40.122041 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.122019 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zrpck\"" Jul 27 21:48:40.122168 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.122105 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 21:48:40.122701 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.122687 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 21:48:40.136336 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.136307 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc"] Jul 27 21:48:40.202651 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.202616 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c4a1941-1263-44a7-9b5c-b9597f76a446-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc\" (UID: \"5c4a1941-1263-44a7-9b5c-b9597f76a446\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" Jul 27 21:48:40.202819 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.202669 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcr4b\" (UniqueName: \"kubernetes.io/projected/5c4a1941-1263-44a7-9b5c-b9597f76a446-kube-api-access-wcr4b\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc\" (UID: \"5c4a1941-1263-44a7-9b5c-b9597f76a446\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" Jul 27 21:48:40.202819 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.202698 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c4a1941-1263-44a7-9b5c-b9597f76a446-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc\" (UID: \"5c4a1941-1263-44a7-9b5c-b9597f76a446\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" Jul 27 21:48:40.304068 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.304031 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c4a1941-1263-44a7-9b5c-b9597f76a446-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc\" (UID: \"5c4a1941-1263-44a7-9b5c-b9597f76a446\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" Jul 27 21:48:40.304216 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.304083 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcr4b\" (UniqueName: \"kubernetes.io/projected/5c4a1941-1263-44a7-9b5c-b9597f76a446-kube-api-access-wcr4b\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc\" (UID: \"5c4a1941-1263-44a7-9b5c-b9597f76a446\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" Jul 27 21:48:40.304291 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.304216 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c4a1941-1263-44a7-9b5c-b9597f76a446-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc\" (UID: \"5c4a1941-1263-44a7-9b5c-b9597f76a446\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" Jul 27 21:48:40.304439 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.304422 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c4a1941-1263-44a7-9b5c-b9597f76a446-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc\" (UID: \"5c4a1941-1263-44a7-9b5c-b9597f76a446\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" Jul 27 21:48:40.304555 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.304539 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c4a1941-1263-44a7-9b5c-b9597f76a446-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc\" (UID: \"5c4a1941-1263-44a7-9b5c-b9597f76a446\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" Jul 27 21:48:40.317672 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.317642 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcr4b\" (UniqueName: \"kubernetes.io/projected/5c4a1941-1263-44a7-9b5c-b9597f76a446-kube-api-access-wcr4b\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc\" (UID: \"5c4a1941-1263-44a7-9b5c-b9597f76a446\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" Jul 27 21:48:40.421388 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.421357 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" Jul 27 21:48:40.562999 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.562961 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc"] Jul 27 21:48:40.565792 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:48:40.565759 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4a1941_1263_44a7_9b5c_b9597f76a446.slice/crio-af3fa2f326647dfae601a8d628f49a76f1d557e7109fcc166a0fc8bedcba377a WatchSource:0}: Error finding container af3fa2f326647dfae601a8d628f49a76f1d557e7109fcc166a0fc8bedcba377a: Status 404 returned error can't find the container with id af3fa2f326647dfae601a8d628f49a76f1d557e7109fcc166a0fc8bedcba377a Jul 27 21:48:40.612251 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:40.612218 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" event={"ID":"5c4a1941-1263-44a7-9b5c-b9597f76a446","Type":"ContainerStarted","Data":"af3fa2f326647dfae601a8d628f49a76f1d557e7109fcc166a0fc8bedcba377a"} Jul 27 21:48:46.958902 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:46.958860 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:47.637984 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:47.637950 2561 generic.go:356] "Generic (PLEG): container finished" podID="5c4a1941-1263-44a7-9b5c-b9597f76a446" containerID="139376c1a0b6971946616ab076f0909d645a8c9a42af7a4be96fbce7b1153c75" exitCode=0 Jul 27 21:48:47.638165 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:47.638008 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" event={"ID":"5c4a1941-1263-44a7-9b5c-b9597f76a446","Type":"ContainerDied","Data":"139376c1a0b6971946616ab076f0909d645a8c9a42af7a4be96fbce7b1153c75"} Jul 27 21:48:50.650373 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:50.650340 2561 generic.go:356] "Generic (PLEG): container finished" podID="5c4a1941-1263-44a7-9b5c-b9597f76a446" containerID="a0e5ca4459c2e4e795c07bf4792ad7a0e5f6463eddb1406366a18054e79f623d" exitCode=0 Jul 27 21:48:50.650728 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:50.650391 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" event={"ID":"5c4a1941-1263-44a7-9b5c-b9597f76a446","Type":"ContainerDied","Data":"a0e5ca4459c2e4e795c07bf4792ad7a0e5f6463eddb1406366a18054e79f623d"} Jul 27 21:48:58.682743 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:58.682711 2561 generic.go:356] "Generic (PLEG): container finished" podID="5c4a1941-1263-44a7-9b5c-b9597f76a446" containerID="33f946fbd7f6016b921776641b47073c0aaef3009099ae5fce7078e68d4ca3ae" exitCode=0 Jul 27 21:48:58.683158 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:58.682789 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" event={"ID":"5c4a1941-1263-44a7-9b5c-b9597f76a446","Type":"ContainerDied","Data":"33f946fbd7f6016b921776641b47073c0aaef3009099ae5fce7078e68d4ca3ae"} Jul 27 21:48:58.863135 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:58.863103 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:59.805692 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:59.805667 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" Jul 27 21:48:59.871022 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:59.870990 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcr4b\" (UniqueName: \"kubernetes.io/projected/5c4a1941-1263-44a7-9b5c-b9597f76a446-kube-api-access-wcr4b\") pod \"5c4a1941-1263-44a7-9b5c-b9597f76a446\" (UID: \"5c4a1941-1263-44a7-9b5c-b9597f76a446\") " Jul 27 21:48:59.871200 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:59.871031 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c4a1941-1263-44a7-9b5c-b9597f76a446-bundle\") pod \"5c4a1941-1263-44a7-9b5c-b9597f76a446\" (UID: \"5c4a1941-1263-44a7-9b5c-b9597f76a446\") " Jul 27 21:48:59.871200 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:59.871061 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c4a1941-1263-44a7-9b5c-b9597f76a446-util\") pod \"5c4a1941-1263-44a7-9b5c-b9597f76a446\" (UID: \"5c4a1941-1263-44a7-9b5c-b9597f76a446\") " Jul 27 21:48:59.871689 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:59.871664 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4a1941-1263-44a7-9b5c-b9597f76a446-bundle" (OuterVolumeSpecName: "bundle") pod "5c4a1941-1263-44a7-9b5c-b9597f76a446" (UID: "5c4a1941-1263-44a7-9b5c-b9597f76a446"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:48:59.873211 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:59.873184 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4a1941-1263-44a7-9b5c-b9597f76a446-kube-api-access-wcr4b" (OuterVolumeSpecName: "kube-api-access-wcr4b") pod "5c4a1941-1263-44a7-9b5c-b9597f76a446" (UID: "5c4a1941-1263-44a7-9b5c-b9597f76a446"). InnerVolumeSpecName "kube-api-access-wcr4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:48:59.876328 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:59.876303 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4a1941-1263-44a7-9b5c-b9597f76a446-util" (OuterVolumeSpecName: "util") pod "5c4a1941-1263-44a7-9b5c-b9597f76a446" (UID: "5c4a1941-1263-44a7-9b5c-b9597f76a446"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:48:59.971910 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:59.971825 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c4a1941-1263-44a7-9b5c-b9597f76a446-util\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:48:59.971910 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:59.971853 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcr4b\" (UniqueName: \"kubernetes.io/projected/5c4a1941-1263-44a7-9b5c-b9597f76a446-kube-api-access-wcr4b\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:48:59.971910 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:48:59.971863 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c4a1941-1263-44a7-9b5c-b9597f76a446-bundle\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:49:00.691104 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:00.691069 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" event={"ID":"5c4a1941-1263-44a7-9b5c-b9597f76a446","Type":"ContainerDied","Data":"af3fa2f326647dfae601a8d628f49a76f1d557e7109fcc166a0fc8bedcba377a"} Jul 27 21:49:00.691283 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:00.691112 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmxpc" Jul 27 21:49:00.691283 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:00.691146 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af3fa2f326647dfae601a8d628f49a76f1d557e7109fcc166a0fc8bedcba377a" Jul 27 21:49:07.532444 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:07.532415 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:07.912105 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:07.912075 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:10.670279 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.670227 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m"] Jul 27 21:49:10.670683 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.670478 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4a1941-1263-44a7-9b5c-b9597f76a446" containerName="pull" Jul 27 21:49:10.670683 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.670488 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4a1941-1263-44a7-9b5c-b9597f76a446" containerName="pull" Jul 27 21:49:10.670683 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.670501 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4a1941-1263-44a7-9b5c-b9597f76a446" containerName="util" Jul 27 21:49:10.670683 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.670507 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4a1941-1263-44a7-9b5c-b9597f76a446" containerName="util" Jul 27 21:49:10.670683 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.670513 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4a1941-1263-44a7-9b5c-b9597f76a446" containerName="extract" Jul 27 21:49:10.670683 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.670518 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4a1941-1263-44a7-9b5c-b9597f76a446" containerName="extract" Jul 27 21:49:10.670683 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.670569 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c4a1941-1263-44a7-9b5c-b9597f76a446" containerName="extract" Jul 27 21:49:10.673865 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.673850 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m" Jul 27 21:49:10.679096 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.679068 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 27 21:49:10.679517 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.679498 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:49:10.680374 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.680355 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9hmt2\"" Jul 27 21:49:10.709665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.709635 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m"] Jul 27 21:49:10.751564 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.751532 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0392f07-b08f-457c-97c6-2341776c23b5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sl78m\" (UID: \"c0392f07-b08f-457c-97c6-2341776c23b5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m" Jul 27 21:49:10.751728 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.751586 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdbzl\" (UniqueName: \"kubernetes.io/projected/c0392f07-b08f-457c-97c6-2341776c23b5-kube-api-access-rdbzl\") pod \"cert-manager-operator-controller-manager-55c444bf86-sl78m\" (UID: \"c0392f07-b08f-457c-97c6-2341776c23b5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m" Jul 27 21:49:10.852314 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.852256 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdbzl\" (UniqueName: \"kubernetes.io/projected/c0392f07-b08f-457c-97c6-2341776c23b5-kube-api-access-rdbzl\") pod \"cert-manager-operator-controller-manager-55c444bf86-sl78m\" (UID: \"c0392f07-b08f-457c-97c6-2341776c23b5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m" Jul 27 21:49:10.852497 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.852353 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0392f07-b08f-457c-97c6-2341776c23b5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sl78m\" (UID: \"c0392f07-b08f-457c-97c6-2341776c23b5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m" Jul 27 21:49:10.852716 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.852699 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0392f07-b08f-457c-97c6-2341776c23b5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sl78m\" (UID: \"c0392f07-b08f-457c-97c6-2341776c23b5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m" Jul 27 21:49:10.864597 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.864577 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdbzl\" (UniqueName: \"kubernetes.io/projected/c0392f07-b08f-457c-97c6-2341776c23b5-kube-api-access-rdbzl\") pod \"cert-manager-operator-controller-manager-55c444bf86-sl78m\" (UID: \"c0392f07-b08f-457c-97c6-2341776c23b5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m" Jul 27 21:49:10.982376 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:10.982248 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m" Jul 27 21:49:11.143307 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:11.143233 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m"] Jul 27 21:49:11.146304 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:49:11.146279 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0392f07_b08f_457c_97c6_2341776c23b5.slice/crio-5383d869bc71a8232050df516c326729a6346c38d70ea6ad0b6c7690e481676c WatchSource:0}: Error finding container 5383d869bc71a8232050df516c326729a6346c38d70ea6ad0b6c7690e481676c: Status 404 returned error can't find the container with id 5383d869bc71a8232050df516c326729a6346c38d70ea6ad0b6c7690e481676c Jul 27 21:49:11.731324 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:11.731292 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m" event={"ID":"c0392f07-b08f-457c-97c6-2341776c23b5","Type":"ContainerStarted","Data":"5383d869bc71a8232050df516c326729a6346c38d70ea6ad0b6c7690e481676c"} Jul 27 21:49:12.097847 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:12.097768 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:12.300509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:12.300466 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:13.740774 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:13.740740 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m" event={"ID":"c0392f07-b08f-457c-97c6-2341776c23b5","Type":"ContainerStarted","Data":"f201f58e24bdf094eaa4329ce1f232500f0bee3db9dee9551758d3ff5fd05f97"} Jul 27 21:49:13.780015 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:13.779966 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl78m" podStartSLOduration=1.959158999 podStartE2EDuration="3.779951827s" totalImagesPullingTime="1.820792828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:49:13.778768814 +0000 UTC m=+231.396206145" watchObservedRunningTime="2026-07-27 21:49:13.779951827 +0000 UTC m=+231.397389145" Jul 27 21:49:14.231858 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.231828 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:14.408372 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.408342 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:14.639802 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.639717 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:14.752804 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.752776 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rt7qz"] Jul 27 21:49:14.754914 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.754896 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rt7qz" Jul 27 21:49:14.757428 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.757404 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-t2z22\"" Jul 27 21:49:14.757986 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.757970 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 27 21:49:14.759050 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.759035 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 27 21:49:14.776590 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.776563 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rt7qz"] Jul 27 21:49:14.784459 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.784433 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5d2\" (UniqueName: \"kubernetes.io/projected/5a06688c-3592-47bc-a278-5332345df7b5-kube-api-access-bt5d2\") pod \"cert-manager-webhook-6486ffd997-rt7qz\" (UID: \"5a06688c-3592-47bc-a278-5332345df7b5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rt7qz" Jul 27 21:49:14.784585 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.784481 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a06688c-3592-47bc-a278-5332345df7b5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rt7qz\" (UID: \"5a06688c-3592-47bc-a278-5332345df7b5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rt7qz" Jul 27 21:49:14.885110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.885075 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a06688c-3592-47bc-a278-5332345df7b5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rt7qz\" (UID: \"5a06688c-3592-47bc-a278-5332345df7b5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rt7qz" Jul 27 21:49:14.885299 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.885129 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5d2\" (UniqueName: \"kubernetes.io/projected/5a06688c-3592-47bc-a278-5332345df7b5-kube-api-access-bt5d2\") pod \"cert-manager-webhook-6486ffd997-rt7qz\" (UID: \"5a06688c-3592-47bc-a278-5332345df7b5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rt7qz" Jul 27 21:49:14.901559 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.901519 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw"] Jul 27 21:49:14.903590 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.903575 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" Jul 27 21:49:14.907435 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.907420 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 21:49:14.907859 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.907845 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zrpck\"" Jul 27 21:49:14.908011 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.907994 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 21:49:14.940184 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.940160 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:14.949839 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.949815 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw"] Jul 27 21:49:14.950597 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.950569 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5d2\" (UniqueName: \"kubernetes.io/projected/5a06688c-3592-47bc-a278-5332345df7b5-kube-api-access-bt5d2\") pod \"cert-manager-webhook-6486ffd997-rt7qz\" (UID: \"5a06688c-3592-47bc-a278-5332345df7b5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rt7qz" Jul 27 21:49:14.950707 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.950571 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a06688c-3592-47bc-a278-5332345df7b5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rt7qz\" (UID: \"5a06688c-3592-47bc-a278-5332345df7b5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rt7qz" Jul 27 21:49:14.985769 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.985736 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d50ca151-4db9-4b04-a268-84519f4b9884-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw\" (UID: \"d50ca151-4db9-4b04-a268-84519f4b9884\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" Jul 27 21:49:14.985769 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.985774 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d50ca151-4db9-4b04-a268-84519f4b9884-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw\" (UID: \"d50ca151-4db9-4b04-a268-84519f4b9884\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" Jul 27 21:49:14.985975 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:14.985815 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmt8l\" (UniqueName: \"kubernetes.io/projected/d50ca151-4db9-4b04-a268-84519f4b9884-kube-api-access-vmt8l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw\" (UID: \"d50ca151-4db9-4b04-a268-84519f4b9884\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" Jul 27 21:49:15.075616 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.075583 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rt7qz" Jul 27 21:49:15.086976 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.086947 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d50ca151-4db9-4b04-a268-84519f4b9884-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw\" (UID: \"d50ca151-4db9-4b04-a268-84519f4b9884\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" Jul 27 21:49:15.087226 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.087199 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmt8l\" (UniqueName: \"kubernetes.io/projected/d50ca151-4db9-4b04-a268-84519f4b9884-kube-api-access-vmt8l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw\" (UID: \"d50ca151-4db9-4b04-a268-84519f4b9884\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" Jul 27 21:49:15.087417 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.087385 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d50ca151-4db9-4b04-a268-84519f4b9884-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw\" (UID: \"d50ca151-4db9-4b04-a268-84519f4b9884\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" Jul 27 21:49:15.087417 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.087401 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d50ca151-4db9-4b04-a268-84519f4b9884-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw\" (UID: \"d50ca151-4db9-4b04-a268-84519f4b9884\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" Jul 27 21:49:15.087722 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.087699 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d50ca151-4db9-4b04-a268-84519f4b9884-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw\" (UID: \"d50ca151-4db9-4b04-a268-84519f4b9884\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" Jul 27 21:49:15.109539 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.109505 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmt8l\" (UniqueName: \"kubernetes.io/projected/d50ca151-4db9-4b04-a268-84519f4b9884-kube-api-access-vmt8l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw\" (UID: \"d50ca151-4db9-4b04-a268-84519f4b9884\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" Jul 27 21:49:15.212927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.212845 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" Jul 27 21:49:15.252870 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.252840 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rt7qz"] Jul 27 21:49:15.256673 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:49:15.256649 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a06688c_3592_47bc_a278_5332345df7b5.slice/crio-15b091053cd79597db86cb3990ff2712a6758d55a738f8e135c2a3baee630f51 WatchSource:0}: Error finding container 15b091053cd79597db86cb3990ff2712a6758d55a738f8e135c2a3baee630f51: Status 404 returned error can't find the container with id 15b091053cd79597db86cb3990ff2712a6758d55a738f8e135c2a3baee630f51 Jul 27 21:49:15.362112 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.362080 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw"] Jul 27 21:49:15.368427 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:49:15.368402 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50ca151_4db9_4b04_a268_84519f4b9884.slice/crio-39ad3c58e29249359b38b19f486fdb43aa7636aff92626033d41e026b5d2d430 WatchSource:0}: Error finding container 39ad3c58e29249359b38b19f486fdb43aa7636aff92626033d41e026b5d2d430: Status 404 returned error can't find the container with id 39ad3c58e29249359b38b19f486fdb43aa7636aff92626033d41e026b5d2d430 Jul 27 21:49:15.749118 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.749079 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rt7qz" event={"ID":"5a06688c-3592-47bc-a278-5332345df7b5","Type":"ContainerStarted","Data":"15b091053cd79597db86cb3990ff2712a6758d55a738f8e135c2a3baee630f51"} Jul 27 21:49:15.750501 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.750476 2561 generic.go:356] "Generic (PLEG): container finished" podID="d50ca151-4db9-4b04-a268-84519f4b9884" containerID="9b68f4623538e05a8794fab278174deaaff7fdaf76262e5bd89b74334b53f5d5" exitCode=0 Jul 27 21:49:15.750644 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.750517 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" event={"ID":"d50ca151-4db9-4b04-a268-84519f4b9884","Type":"ContainerDied","Data":"9b68f4623538e05a8794fab278174deaaff7fdaf76262e5bd89b74334b53f5d5"} Jul 27 21:49:15.750644 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:15.750546 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" event={"ID":"d50ca151-4db9-4b04-a268-84519f4b9884","Type":"ContainerStarted","Data":"39ad3c58e29249359b38b19f486fdb43aa7636aff92626033d41e026b5d2d430"} Jul 27 21:49:16.642537 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:16.642502 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:16.839129 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:16.839053 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:18.497554 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:18.497513 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:18.648938 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:18.648904 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:19.771441 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:19.771404 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rt7qz" event={"ID":"5a06688c-3592-47bc-a278-5332345df7b5","Type":"ContainerStarted","Data":"9c17f6665829b0d36c5d7339db46b6fceba68b596aabbbb511cb1891b94f338d"} Jul 27 21:49:19.771852 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:19.771457 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rt7qz" Jul 27 21:49:19.773098 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:19.773073 2561 generic.go:356] "Generic (PLEG): container finished" podID="d50ca151-4db9-4b04-a268-84519f4b9884" containerID="7097db83ddc0d67393d9213c3786fc4e36a89f043595d18f50315b2233afb7b4" exitCode=0 Jul 27 21:49:19.773192 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:19.773100 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" event={"ID":"d50ca151-4db9-4b04-a268-84519f4b9884","Type":"ContainerDied","Data":"7097db83ddc0d67393d9213c3786fc4e36a89f043595d18f50315b2233afb7b4"} Jul 27 21:49:19.808366 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:19.808321 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-rt7qz" podStartSLOduration=2.074690923 podStartE2EDuration="5.808306563s" totalImagesPullingTime="3.73361564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:49:19.80737611 +0000 UTC m=+237.424813429" watchObservedRunningTime="2026-07-27 21:49:19.808306563 +0000 UTC m=+237.425743881" Jul 27 21:49:20.778627 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:20.778597 2561 generic.go:356] "Generic (PLEG): container finished" podID="d50ca151-4db9-4b04-a268-84519f4b9884" containerID="211644611e65a0dc0b043a6358438edd0df3f17c2c8817b242bb7d3ab4662924" exitCode=0 Jul 27 21:49:20.779012 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:20.778669 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" event={"ID":"d50ca151-4db9-4b04-a268-84519f4b9884","Type":"ContainerDied","Data":"211644611e65a0dc0b043a6358438edd0df3f17c2c8817b242bb7d3ab4662924"} Jul 27 21:49:21.904789 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:21.904766 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" Jul 27 21:49:22.043186 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:22.043100 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmt8l\" (UniqueName: \"kubernetes.io/projected/d50ca151-4db9-4b04-a268-84519f4b9884-kube-api-access-vmt8l\") pod \"d50ca151-4db9-4b04-a268-84519f4b9884\" (UID: \"d50ca151-4db9-4b04-a268-84519f4b9884\") " Jul 27 21:49:22.043186 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:22.043163 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d50ca151-4db9-4b04-a268-84519f4b9884-util\") pod \"d50ca151-4db9-4b04-a268-84519f4b9884\" (UID: \"d50ca151-4db9-4b04-a268-84519f4b9884\") " Jul 27 21:49:22.043186 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:22.043179 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d50ca151-4db9-4b04-a268-84519f4b9884-bundle\") pod \"d50ca151-4db9-4b04-a268-84519f4b9884\" (UID: \"d50ca151-4db9-4b04-a268-84519f4b9884\") " Jul 27 21:49:22.043632 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:22.043606 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50ca151-4db9-4b04-a268-84519f4b9884-bundle" (OuterVolumeSpecName: "bundle") pod "d50ca151-4db9-4b04-a268-84519f4b9884" (UID: "d50ca151-4db9-4b04-a268-84519f4b9884"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:49:22.045169 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:22.045147 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50ca151-4db9-4b04-a268-84519f4b9884-kube-api-access-vmt8l" (OuterVolumeSpecName: "kube-api-access-vmt8l") pod "d50ca151-4db9-4b04-a268-84519f4b9884" (UID: "d50ca151-4db9-4b04-a268-84519f4b9884"). InnerVolumeSpecName "kube-api-access-vmt8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:49:22.047413 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:22.047369 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50ca151-4db9-4b04-a268-84519f4b9884-util" (OuterVolumeSpecName: "util") pod "d50ca151-4db9-4b04-a268-84519f4b9884" (UID: "d50ca151-4db9-4b04-a268-84519f4b9884"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:49:22.144355 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:22.144323 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmt8l\" (UniqueName: \"kubernetes.io/projected/d50ca151-4db9-4b04-a268-84519f4b9884-kube-api-access-vmt8l\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:49:22.144355 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:22.144352 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d50ca151-4db9-4b04-a268-84519f4b9884-util\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:49:22.144355 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:22.144364 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d50ca151-4db9-4b04-a268-84519f4b9884-bundle\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:49:22.787712 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:22.787682 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" Jul 27 21:49:22.787712 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:22.787693 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm8sgw" event={"ID":"d50ca151-4db9-4b04-a268-84519f4b9884","Type":"ContainerDied","Data":"39ad3c58e29249359b38b19f486fdb43aa7636aff92626033d41e026b5d2d430"} Jul 27 21:49:22.787905 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:22.787761 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39ad3c58e29249359b38b19f486fdb43aa7636aff92626033d41e026b5d2d430" Jul 27 21:49:23.820925 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:23.820889 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:25.302129 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:25.302097 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:25.448738 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:25.448704 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:25.780539 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:25.780508 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rt7qz" Jul 27 21:49:30.740478 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:30.740431 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:30.836798 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:30.836766 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:32.437746 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.437712 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8"] Jul 27 21:49:32.438118 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.437967 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50ca151-4db9-4b04-a268-84519f4b9884" containerName="pull" Jul 27 21:49:32.438118 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.437976 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50ca151-4db9-4b04-a268-84519f4b9884" containerName="pull" Jul 27 21:49:32.438118 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.437988 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50ca151-4db9-4b04-a268-84519f4b9884" containerName="extract" Jul 27 21:49:32.438118 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.437994 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50ca151-4db9-4b04-a268-84519f4b9884" containerName="extract" Jul 27 21:49:32.438118 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.438004 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50ca151-4db9-4b04-a268-84519f4b9884" containerName="util" Jul 27 21:49:32.438118 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.438009 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50ca151-4db9-4b04-a268-84519f4b9884" containerName="util" Jul 27 21:49:32.438118 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.438048 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50ca151-4db9-4b04-a268-84519f4b9884" containerName="extract" Jul 27 21:49:32.485094 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.485059 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8"] Jul 27 21:49:32.485250 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.485171 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" Jul 27 21:49:32.487570 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.487542 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 21:49:32.487714 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.487542 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zrpck\"" Jul 27 21:49:32.487714 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.487599 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 21:49:32.555897 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.555870 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:32.626173 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.626139 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0268516c-39d1-4ec8-9057-ccb016011c94-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8\" (UID: \"0268516c-39d1-4ec8-9057-ccb016011c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" Jul 27 21:49:32.626173 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.626179 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m6ww\" (UniqueName: \"kubernetes.io/projected/0268516c-39d1-4ec8-9057-ccb016011c94-kube-api-access-2m6ww\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8\" (UID: \"0268516c-39d1-4ec8-9057-ccb016011c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" Jul 27 21:49:32.626422 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.626278 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0268516c-39d1-4ec8-9057-ccb016011c94-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8\" (UID: \"0268516c-39d1-4ec8-9057-ccb016011c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" Jul 27 21:49:32.726986 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.726902 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m6ww\" (UniqueName: \"kubernetes.io/projected/0268516c-39d1-4ec8-9057-ccb016011c94-kube-api-access-2m6ww\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8\" (UID: \"0268516c-39d1-4ec8-9057-ccb016011c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" Jul 27 21:49:32.727145 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.727007 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0268516c-39d1-4ec8-9057-ccb016011c94-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8\" (UID: \"0268516c-39d1-4ec8-9057-ccb016011c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" Jul 27 21:49:32.727145 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.727040 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0268516c-39d1-4ec8-9057-ccb016011c94-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8\" (UID: \"0268516c-39d1-4ec8-9057-ccb016011c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" Jul 27 21:49:32.727450 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.727429 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0268516c-39d1-4ec8-9057-ccb016011c94-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8\" (UID: \"0268516c-39d1-4ec8-9057-ccb016011c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" Jul 27 21:49:32.727524 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.727466 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0268516c-39d1-4ec8-9057-ccb016011c94-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8\" (UID: \"0268516c-39d1-4ec8-9057-ccb016011c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" Jul 27 21:49:32.735672 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.735639 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m6ww\" (UniqueName: \"kubernetes.io/projected/0268516c-39d1-4ec8-9057-ccb016011c94-kube-api-access-2m6ww\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8\" (UID: \"0268516c-39d1-4ec8-9057-ccb016011c94\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" Jul 27 21:49:32.794554 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.794523 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" Jul 27 21:49:32.943282 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:32.943228 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8"] Jul 27 21:49:32.946400 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:49:32.946379 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0268516c_39d1_4ec8_9057_ccb016011c94.slice/crio-aab9211cb8b8d764da75a09e195700394cb0185e6cf2ca8c8bf0dd874818393b WatchSource:0}: Error finding container aab9211cb8b8d764da75a09e195700394cb0185e6cf2ca8c8bf0dd874818393b: Status 404 returned error can't find the container with id aab9211cb8b8d764da75a09e195700394cb0185e6cf2ca8c8bf0dd874818393b Jul 27 21:49:33.836673 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:33.836635 2561 generic.go:356] "Generic (PLEG): container finished" podID="0268516c-39d1-4ec8-9057-ccb016011c94" containerID="2bd5ce7070f9d2bcedcde43dcd04f59cde0dfcaba2082fd3541984215510cc1b" exitCode=0 Jul 27 21:49:33.837065 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:33.836720 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" event={"ID":"0268516c-39d1-4ec8-9057-ccb016011c94","Type":"ContainerDied","Data":"2bd5ce7070f9d2bcedcde43dcd04f59cde0dfcaba2082fd3541984215510cc1b"} Jul 27 21:49:33.837065 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:33.836817 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" event={"ID":"0268516c-39d1-4ec8-9057-ccb016011c94","Type":"ContainerStarted","Data":"aab9211cb8b8d764da75a09e195700394cb0185e6cf2ca8c8bf0dd874818393b"} Jul 27 21:49:35.845926 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:35.845893 2561 generic.go:356] "Generic (PLEG): container finished" podID="0268516c-39d1-4ec8-9057-ccb016011c94" containerID="541436641ecf2bf4e1d511f9c89508852287dffd381a62807091afac763765a7" exitCode=0 Jul 27 21:49:35.846375 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:35.845955 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" event={"ID":"0268516c-39d1-4ec8-9057-ccb016011c94","Type":"ContainerDied","Data":"541436641ecf2bf4e1d511f9c89508852287dffd381a62807091afac763765a7"} Jul 27 21:49:36.851650 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:36.851619 2561 generic.go:356] "Generic (PLEG): container finished" podID="0268516c-39d1-4ec8-9057-ccb016011c94" containerID="c177e971efa4008b8a90f85d9e9977d99eb3917a7d47cb691d40e27992d90dc2" exitCode=0 Jul 27 21:49:36.852035 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:36.851684 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" event={"ID":"0268516c-39d1-4ec8-9057-ccb016011c94","Type":"ContainerDied","Data":"c177e971efa4008b8a90f85d9e9977d99eb3917a7d47cb691d40e27992d90dc2"} Jul 27 21:49:37.983382 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:37.983358 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" Jul 27 21:49:38.066242 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:38.066209 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0268516c-39d1-4ec8-9057-ccb016011c94-bundle\") pod \"0268516c-39d1-4ec8-9057-ccb016011c94\" (UID: \"0268516c-39d1-4ec8-9057-ccb016011c94\") " Jul 27 21:49:38.066456 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:38.066288 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0268516c-39d1-4ec8-9057-ccb016011c94-util\") pod \"0268516c-39d1-4ec8-9057-ccb016011c94\" (UID: \"0268516c-39d1-4ec8-9057-ccb016011c94\") " Jul 27 21:49:38.066456 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:38.066331 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m6ww\" (UniqueName: \"kubernetes.io/projected/0268516c-39d1-4ec8-9057-ccb016011c94-kube-api-access-2m6ww\") pod \"0268516c-39d1-4ec8-9057-ccb016011c94\" (UID: \"0268516c-39d1-4ec8-9057-ccb016011c94\") " Jul 27 21:49:38.066976 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:38.066950 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0268516c-39d1-4ec8-9057-ccb016011c94-bundle" (OuterVolumeSpecName: "bundle") pod "0268516c-39d1-4ec8-9057-ccb016011c94" (UID: "0268516c-39d1-4ec8-9057-ccb016011c94"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:49:38.068474 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:38.068450 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0268516c-39d1-4ec8-9057-ccb016011c94-kube-api-access-2m6ww" (OuterVolumeSpecName: "kube-api-access-2m6ww") pod "0268516c-39d1-4ec8-9057-ccb016011c94" (UID: "0268516c-39d1-4ec8-9057-ccb016011c94"). InnerVolumeSpecName "kube-api-access-2m6ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:49:38.071893 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:38.071852 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0268516c-39d1-4ec8-9057-ccb016011c94-util" (OuterVolumeSpecName: "util") pod "0268516c-39d1-4ec8-9057-ccb016011c94" (UID: "0268516c-39d1-4ec8-9057-ccb016011c94"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:49:38.167558 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:38.167522 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0268516c-39d1-4ec8-9057-ccb016011c94-bundle\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:49:38.167558 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:38.167552 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0268516c-39d1-4ec8-9057-ccb016011c94-util\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:49:38.167558 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:38.167564 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m6ww\" (UniqueName: \"kubernetes.io/projected/0268516c-39d1-4ec8-9057-ccb016011c94-kube-api-access-2m6ww\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:49:38.861497 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:38.861473 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" Jul 27 21:49:38.861497 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:38.861485 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6bd8" event={"ID":"0268516c-39d1-4ec8-9057-ccb016011c94","Type":"ContainerDied","Data":"aab9211cb8b8d764da75a09e195700394cb0185e6cf2ca8c8bf0dd874818393b"} Jul 27 21:49:38.861677 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:38.861523 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aab9211cb8b8d764da75a09e195700394cb0185e6cf2ca8c8bf0dd874818393b" Jul 27 21:49:44.751083 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:44.750977 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:44.851530 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:44.851483 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:45.356570 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.356539 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:45.632026 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.631939 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:45.739953 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.739921 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:45.773302 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.773270 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n"] Jul 27 21:49:45.773676 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.773542 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0268516c-39d1-4ec8-9057-ccb016011c94" containerName="util" Jul 27 21:49:45.773676 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.773554 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0268516c-39d1-4ec8-9057-ccb016011c94" containerName="util" Jul 27 21:49:45.773676 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.773561 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0268516c-39d1-4ec8-9057-ccb016011c94" containerName="pull" Jul 27 21:49:45.773676 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.773566 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0268516c-39d1-4ec8-9057-ccb016011c94" containerName="pull" Jul 27 21:49:45.773676 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.773580 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0268516c-39d1-4ec8-9057-ccb016011c94" containerName="extract" Jul 27 21:49:45.773676 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.773586 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0268516c-39d1-4ec8-9057-ccb016011c94" containerName="extract" Jul 27 21:49:45.773676 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.773630 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0268516c-39d1-4ec8-9057-ccb016011c94" containerName="extract" Jul 27 21:49:45.775509 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.775492 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:45.808818 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.808787 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 27 21:49:45.808991 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.808828 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 27 21:49:45.810041 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.810024 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 27 21:49:45.810100 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.810030 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 27 21:49:45.817114 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.817096 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jbjdz\"" Jul 27 21:49:45.859707 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.859681 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n"] Jul 27 21:49:45.923704 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.923669 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9jm4\" (UniqueName: \"kubernetes.io/projected/52d414f2-e4c1-4f85-a815-58fc04fbd99c-kube-api-access-p9jm4\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:45.924009 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.923989 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52d414f2-e4c1-4f85-a815-58fc04fbd99c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:45.924103 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.924091 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52d414f2-e4c1-4f85-a815-58fc04fbd99c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:45.924206 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.924193 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52d414f2-e4c1-4f85-a815-58fc04fbd99c-manifests\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:45.924316 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:45.924303 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52d414f2-e4c1-4f85-a815-58fc04fbd99c-tmp\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:46.025817 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.025783 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9jm4\" (UniqueName: \"kubernetes.io/projected/52d414f2-e4c1-4f85-a815-58fc04fbd99c-kube-api-access-p9jm4\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:46.026034 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.025834 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52d414f2-e4c1-4f85-a815-58fc04fbd99c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:46.026034 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.025852 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52d414f2-e4c1-4f85-a815-58fc04fbd99c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:46.026034 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.025877 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52d414f2-e4c1-4f85-a815-58fc04fbd99c-manifests\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:46.026034 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.025898 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52d414f2-e4c1-4f85-a815-58fc04fbd99c-tmp\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:46.026404 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.026377 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52d414f2-e4c1-4f85-a815-58fc04fbd99c-manifests\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:46.028244 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.028218 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52d414f2-e4c1-4f85-a815-58fc04fbd99c-tmp\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:46.028419 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.028402 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52d414f2-e4c1-4f85-a815-58fc04fbd99c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:46.028472 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.028419 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52d414f2-e4c1-4f85-a815-58fc04fbd99c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:46.042299 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.042256 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9jm4\" (UniqueName: \"kubernetes.io/projected/52d414f2-e4c1-4f85-a815-58fc04fbd99c-kube-api-access-p9jm4\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n\" (UID: \"52d414f2-e4c1-4f85-a815-58fc04fbd99c\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:46.084977 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.084933 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:46.117708 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.117680 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:46.227708 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.227473 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:46.228686 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.228659 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n"] Jul 27 21:49:46.232388 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:49:46.232366 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d414f2_e4c1_4f85_a815_58fc04fbd99c.slice/crio-5ff5c28c9aa961cd8c6d4e94553c93c11c763c7a1ef12e01b27e033f25489b16 WatchSource:0}: Error finding container 5ff5c28c9aa961cd8c6d4e94553c93c11c763c7a1ef12e01b27e033f25489b16: Status 404 returned error can't find the container with id 5ff5c28c9aa961cd8c6d4e94553c93c11c763c7a1ef12e01b27e033f25489b16 Jul 27 21:49:46.344730 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.344693 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:46.446420 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.446386 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:46.576710 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.576631 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:46.896024 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:46.895942 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" event={"ID":"52d414f2-e4c1-4f85-a815-58fc04fbd99c","Type":"ContainerStarted","Data":"5ff5c28c9aa961cd8c6d4e94553c93c11c763c7a1ef12e01b27e033f25489b16"} Jul 27 21:49:49.911891 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:49.911860 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" event={"ID":"52d414f2-e4c1-4f85-a815-58fc04fbd99c","Type":"ContainerStarted","Data":"f2b79fd89072a04bdc1c820e34ed86ae909c610c16d89fb9a0f3d28a4ee46084"} Jul 27 21:49:50.917131 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:50.917099 2561 generic.go:356] "Generic (PLEG): container finished" podID="52d414f2-e4c1-4f85-a815-58fc04fbd99c" containerID="f2b79fd89072a04bdc1c820e34ed86ae909c610c16d89fb9a0f3d28a4ee46084" exitCode=0 Jul 27 21:49:50.917534 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:50.917173 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" event={"ID":"52d414f2-e4c1-4f85-a815-58fc04fbd99c","Type":"ContainerDied","Data":"f2b79fd89072a04bdc1c820e34ed86ae909c610c16d89fb9a0f3d28a4ee46084"} Jul 27 21:49:51.813058 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:51.813026 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:51.922743 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:51.922702 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" event={"ID":"52d414f2-e4c1-4f85-a815-58fc04fbd99c","Type":"ContainerStarted","Data":"64795b8877edfea24ac316632f012d692f7a8bf207e2e8f3754fe3ce8f8fbfe8"} Jul 27 21:49:51.923113 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:51.922840 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:49:51.971317 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:51.971251 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" podStartSLOduration=2.608118962 podStartE2EDuration="6.971234524s" totalImagesPullingTime="3.271498048s" totalInitContainerRuntime="1.091617514s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:49:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:49:51.969158323 +0000 UTC m=+269.586595643" watchObservedRunningTime="2026-07-27 21:49:51.971234524 +0000 UTC m=+269.588671842" Jul 27 21:49:51.974873 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:51.974846 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:56.785028 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:56.784993 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:49:56.886833 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:49:56.886802 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:02.928937 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:02.928905 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n" Jul 27 21:50:04.940678 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:04.940645 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:05.448210 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.448175 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs"] Jul 27 21:50:05.456475 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.456456 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.460475 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.460447 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 27 21:50:05.460475 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.460473 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 27 21:50:05.460726 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.460501 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rpm89\"" Jul 27 21:50:05.460726 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.460627 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 27 21:50:05.460726 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.460647 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 27 21:50:05.460899 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.460884 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:50:05.467940 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.467916 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs"] Jul 27 21:50:05.478979 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.478949 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e73b8505-9c44-40f5-a50a-297a426ebb3e-manager-config\") pod \"lws-controller-manager-647fb87ffb-7tdxs\" (UID: \"e73b8505-9c44-40f5-a50a-297a426ebb3e\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.479105 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.478997 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e73b8505-9c44-40f5-a50a-297a426ebb3e-metrics-cert\") pod \"lws-controller-manager-647fb87ffb-7tdxs\" (UID: \"e73b8505-9c44-40f5-a50a-297a426ebb3e\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.479105 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.479025 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e73b8505-9c44-40f5-a50a-297a426ebb3e-cert\") pod \"lws-controller-manager-647fb87ffb-7tdxs\" (UID: \"e73b8505-9c44-40f5-a50a-297a426ebb3e\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.479105 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.479049 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78n59\" (UniqueName: \"kubernetes.io/projected/e73b8505-9c44-40f5-a50a-297a426ebb3e-kube-api-access-78n59\") pod \"lws-controller-manager-647fb87ffb-7tdxs\" (UID: \"e73b8505-9c44-40f5-a50a-297a426ebb3e\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.579469 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.579432 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e73b8505-9c44-40f5-a50a-297a426ebb3e-metrics-cert\") pod \"lws-controller-manager-647fb87ffb-7tdxs\" (UID: \"e73b8505-9c44-40f5-a50a-297a426ebb3e\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.579664 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.579481 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e73b8505-9c44-40f5-a50a-297a426ebb3e-cert\") pod \"lws-controller-manager-647fb87ffb-7tdxs\" (UID: \"e73b8505-9c44-40f5-a50a-297a426ebb3e\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.579664 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.579598 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78n59\" (UniqueName: \"kubernetes.io/projected/e73b8505-9c44-40f5-a50a-297a426ebb3e-kube-api-access-78n59\") pod \"lws-controller-manager-647fb87ffb-7tdxs\" (UID: \"e73b8505-9c44-40f5-a50a-297a426ebb3e\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.579786 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.579681 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e73b8505-9c44-40f5-a50a-297a426ebb3e-manager-config\") pod \"lws-controller-manager-647fb87ffb-7tdxs\" (UID: \"e73b8505-9c44-40f5-a50a-297a426ebb3e\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.580348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.580326 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e73b8505-9c44-40f5-a50a-297a426ebb3e-manager-config\") pod \"lws-controller-manager-647fb87ffb-7tdxs\" (UID: \"e73b8505-9c44-40f5-a50a-297a426ebb3e\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.581903 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.581877 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e73b8505-9c44-40f5-a50a-297a426ebb3e-metrics-cert\") pod \"lws-controller-manager-647fb87ffb-7tdxs\" (UID: \"e73b8505-9c44-40f5-a50a-297a426ebb3e\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.582045 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.582024 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e73b8505-9c44-40f5-a50a-297a426ebb3e-cert\") pod \"lws-controller-manager-647fb87ffb-7tdxs\" (UID: \"e73b8505-9c44-40f5-a50a-297a426ebb3e\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.594596 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.594571 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78n59\" (UniqueName: \"kubernetes.io/projected/e73b8505-9c44-40f5-a50a-297a426ebb3e-kube-api-access-78n59\") pod \"lws-controller-manager-647fb87ffb-7tdxs\" (UID: \"e73b8505-9c44-40f5-a50a-297a426ebb3e\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.766433 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.766353 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:05.905417 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.905298 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs"] Jul 27 21:50:05.909590 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:50:05.909567 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73b8505_9c44_40f5_a50a_297a426ebb3e.slice/crio-ae8f1d9580cee3f049ecea8b8824b1fda789963937aec4d62aee3001667f2e5d WatchSource:0}: Error finding container ae8f1d9580cee3f049ecea8b8824b1fda789963937aec4d62aee3001667f2e5d: Status 404 returned error can't find the container with id ae8f1d9580cee3f049ecea8b8824b1fda789963937aec4d62aee3001667f2e5d Jul 27 21:50:05.980890 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:05.980856 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" event={"ID":"e73b8505-9c44-40f5-a50a-297a426ebb3e","Type":"ContainerStarted","Data":"ae8f1d9580cee3f049ecea8b8824b1fda789963937aec4d62aee3001667f2e5d"} Jul 27 21:50:06.670079 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:06.670040 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:07.991158 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:07.991124 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" event={"ID":"e73b8505-9c44-40f5-a50a-297a426ebb3e","Type":"ContainerStarted","Data":"0e916f00dff6ba7321aae61b61998b10b0c51b588ebf3eb4c7c81bf9a68ea846"} Jul 27 21:50:07.991546 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:07.991182 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:08.009832 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:08.009787 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" podStartSLOduration=1.393750705 podStartE2EDuration="3.0097723s" totalImagesPullingTime="1.616021595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:50:08.008650799 +0000 UTC m=+285.626088118" watchObservedRunningTime="2026-07-27 21:50:08.0097723 +0000 UTC m=+285.627209618" Jul 27 21:50:16.021542 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:16.021509 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:16.117279 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:16.117231 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:16.227709 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:16.227669 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:16.445304 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:16.445249 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:16.739293 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:16.739190 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:16.884426 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:16.884394 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:17.369852 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:17.369817 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:17.487557 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:17.487528 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:18.570215 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:18.570186 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:18.668113 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:18.668079 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:18.997665 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:18.997638 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 21:50:19.902655 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:19.902627 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:22.738796 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:22.738766 2561 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 21:50:23.491120 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:23.491085 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:23.597087 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:23.597058 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:24.148436 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.148406 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b"] Jul 27 21:50:24.157423 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.157400 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.159751 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.159719 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 27 21:50:24.159869 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.159753 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-pjk29\"" Jul 27 21:50:24.165088 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.165064 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b"] Jul 27 21:50:24.191007 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.190975 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:24.223317 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.223284 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b40f45-4e8e-4638-8858-c9c8114baeef-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.223505 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.223323 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.223505 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.223405 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.223505 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.223423 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b40f45-4e8e-4638-8858-c9c8114baeef-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.223505 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.223479 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.223733 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.223527 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zmm4\" (UniqueName: \"kubernetes.io/projected/b1b40f45-4e8e-4638-8858-c9c8114baeef-kube-api-access-2zmm4\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.223733 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.223545 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.223733 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.223560 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.223733 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.223578 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b40f45-4e8e-4638-8858-c9c8114baeef-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.324650 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.324592 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.324650 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.324657 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b40f45-4e8e-4638-8858-c9c8114baeef-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.324878 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.324692 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.324878 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.324733 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zmm4\" (UniqueName: \"kubernetes.io/projected/b1b40f45-4e8e-4638-8858-c9c8114baeef-kube-api-access-2zmm4\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.324878 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.324761 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.325022 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.324935 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.325022 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.324978 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b40f45-4e8e-4638-8858-c9c8114baeef-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.325147 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.325063 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.325147 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.325068 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b40f45-4e8e-4638-8858-c9c8114baeef-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.325147 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.325127 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.325324 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.325292 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.325896 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.325870 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b40f45-4e8e-4638-8858-c9c8114baeef-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.326030 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.326011 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.326190 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.326168 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.327174 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.327148 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b40f45-4e8e-4638-8858-c9c8114baeef-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.327420 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.327400 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b40f45-4e8e-4638-8858-c9c8114baeef-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.333537 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.333504 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b40f45-4e8e-4638-8858-c9c8114baeef-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.333660 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.333566 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zmm4\" (UniqueName: \"kubernetes.io/projected/b1b40f45-4e8e-4638-8858-c9c8114baeef-kube-api-access-2zmm4\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cszb2b\" (UID: \"b1b40f45-4e8e-4638-8858-c9c8114baeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.468661 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.468565 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:24.605715 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.605689 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b"] Jul 27 21:50:24.607999 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:50:24.607974 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b40f45_4e8e_4638_8858_c9c8114baeef.slice/crio-f5be9993252ea7324f4f5abfa292e08f431d9d81d010d02a37747cb9a2c592a4 WatchSource:0}: Error finding container f5be9993252ea7324f4f5abfa292e08f431d9d81d010d02a37747cb9a2c592a4: Status 404 returned error can't find the container with id f5be9993252ea7324f4f5abfa292e08f431d9d81d010d02a37747cb9a2c592a4 Jul 27 21:50:24.610087 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.610065 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 21:50:24.671905 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.671875 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:24.773075 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:24.772983 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:25.058938 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:25.058853 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" event={"ID":"b1b40f45-4e8e-4638-8858-c9c8114baeef","Type":"ContainerStarted","Data":"f5be9993252ea7324f4f5abfa292e08f431d9d81d010d02a37747cb9a2c592a4"} Jul 27 21:50:27.292141 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:27.292103 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 21:50:27.292421 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:27.292179 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 21:50:27.292421 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:27.292206 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 21:50:28.074181 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:28.074144 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" event={"ID":"b1b40f45-4e8e-4638-8858-c9c8114baeef","Type":"ContainerStarted","Data":"c31933302cb829cf1c26013666f98f0a4053cd22330b0379a1b65690ccfeb50c"} Jul 27 21:50:28.096591 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:28.096420 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" podStartSLOduration=1.414825607 podStartE2EDuration="4.096399568s" totalImagesPullingTime="2.681573961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:50:28.095864083 +0000 UTC m=+305.713301403" watchObservedRunningTime="2026-07-27 21:50:28.096399568 +0000 UTC m=+305.713836889" Jul 27 21:50:28.468956 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:28.468923 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:28.473666 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:28.473642 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:29.078400 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:29.078365 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:29.079369 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:29.079348 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cszb2b" Jul 27 21:50:50.700088 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:50:50.700052 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:06.565551 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:06.565514 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:06.661917 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:06.661874 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:06.691102 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:06.691068 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-khdc6"] Jul 27 21:51:06.700589 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:06.700555 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-khdc6" Jul 27 21:51:06.702939 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:06.702911 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 21:51:06.703235 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:06.703212 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-f5rtk\"" Jul 27 21:51:06.703363 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:06.703339 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 21:51:06.704447 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:06.704423 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-khdc6"] Jul 27 21:51:06.763213 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:06.763178 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:06.768315 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:06.768284 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ng6w\" (UniqueName: \"kubernetes.io/projected/055b597f-59a9-4d71-b9ef-81c3033a988f-kube-api-access-7ng6w\") pod \"kuadrant-operator-catalog-khdc6\" (UID: \"055b597f-59a9-4d71-b9ef-81c3033a988f\") " pod="kuadrant-system/kuadrant-operator-catalog-khdc6" Jul 27 21:51:06.869416 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:06.869332 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ng6w\" (UniqueName: \"kubernetes.io/projected/055b597f-59a9-4d71-b9ef-81c3033a988f-kube-api-access-7ng6w\") pod \"kuadrant-operator-catalog-khdc6\" (UID: \"055b597f-59a9-4d71-b9ef-81c3033a988f\") " pod="kuadrant-system/kuadrant-operator-catalog-khdc6" Jul 27 21:51:06.878711 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:06.878678 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ng6w\" (UniqueName: \"kubernetes.io/projected/055b597f-59a9-4d71-b9ef-81c3033a988f-kube-api-access-7ng6w\") pod \"kuadrant-operator-catalog-khdc6\" (UID: \"055b597f-59a9-4d71-b9ef-81c3033a988f\") " pod="kuadrant-system/kuadrant-operator-catalog-khdc6" Jul 27 21:51:07.013228 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:07.013193 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-khdc6" Jul 27 21:51:07.056935 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:07.056900 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-khdc6"] Jul 27 21:51:07.142503 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:07.142480 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-khdc6"] Jul 27 21:51:07.145320 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:51:07.145292 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055b597f_59a9_4d71_b9ef_81c3033a988f.slice/crio-5f0f33e30c787ea194b0248114681641735871bd89d483964b19bc436df09f5c WatchSource:0}: Error finding container 5f0f33e30c787ea194b0248114681641735871bd89d483964b19bc436df09f5c: Status 404 returned error can't find the container with id 5f0f33e30c787ea194b0248114681641735871bd89d483964b19bc436df09f5c Jul 27 21:51:07.235082 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:07.235047 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-khdc6" event={"ID":"055b597f-59a9-4d71-b9ef-81c3033a988f","Type":"ContainerStarted","Data":"5f0f33e30c787ea194b0248114681641735871bd89d483964b19bc436df09f5c"} Jul 27 21:51:07.268736 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:07.268704 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k9xnl"] Jul 27 21:51:07.273583 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:07.273568 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k9xnl" Jul 27 21:51:07.281172 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:07.281149 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k9xnl"] Jul 27 21:51:07.373125 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:07.373093 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kdlh\" (UniqueName: \"kubernetes.io/projected/57c2f4c0-758c-4ff9-9e75-2437780c3f62-kube-api-access-8kdlh\") pod \"kuadrant-operator-catalog-k9xnl\" (UID: \"57c2f4c0-758c-4ff9-9e75-2437780c3f62\") " pod="kuadrant-system/kuadrant-operator-catalog-k9xnl" Jul 27 21:51:07.473937 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:07.473900 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kdlh\" (UniqueName: \"kubernetes.io/projected/57c2f4c0-758c-4ff9-9e75-2437780c3f62-kube-api-access-8kdlh\") pod \"kuadrant-operator-catalog-k9xnl\" (UID: \"57c2f4c0-758c-4ff9-9e75-2437780c3f62\") " pod="kuadrant-system/kuadrant-operator-catalog-k9xnl" Jul 27 21:51:07.482027 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:07.482001 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kdlh\" (UniqueName: \"kubernetes.io/projected/57c2f4c0-758c-4ff9-9e75-2437780c3f62-kube-api-access-8kdlh\") pod \"kuadrant-operator-catalog-k9xnl\" (UID: \"57c2f4c0-758c-4ff9-9e75-2437780c3f62\") " pod="kuadrant-system/kuadrant-operator-catalog-k9xnl" Jul 27 21:51:07.583376 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:07.583337 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k9xnl" Jul 27 21:51:07.717516 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:07.715039 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k9xnl"] Jul 27 21:51:07.717516 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:51:07.717188 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c2f4c0_758c_4ff9_9e75_2437780c3f62.slice/crio-ef0a490e5ff0810d5bbec26f0e8b1bf64e4aa5c8a0762a468b1b9f35f8ee2d59 WatchSource:0}: Error finding container ef0a490e5ff0810d5bbec26f0e8b1bf64e4aa5c8a0762a468b1b9f35f8ee2d59: Status 404 returned error can't find the container with id ef0a490e5ff0810d5bbec26f0e8b1bf64e4aa5c8a0762a468b1b9f35f8ee2d59 Jul 27 21:51:08.240937 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:08.240900 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k9xnl" event={"ID":"57c2f4c0-758c-4ff9-9e75-2437780c3f62","Type":"ContainerStarted","Data":"ef0a490e5ff0810d5bbec26f0e8b1bf64e4aa5c8a0762a468b1b9f35f8ee2d59"} Jul 27 21:51:10.251605 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:10.251563 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-khdc6" event={"ID":"055b597f-59a9-4d71-b9ef-81c3033a988f","Type":"ContainerStarted","Data":"e3cc17d9211540f3eded1c85aef5a683dd49037f265d0d3ca4e1e9902d3ba170"} Jul 27 21:51:10.252042 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:10.251642 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-khdc6" podUID="055b597f-59a9-4d71-b9ef-81c3033a988f" containerName="registry-server" containerID="cri-o://e3cc17d9211540f3eded1c85aef5a683dd49037f265d0d3ca4e1e9902d3ba170" gracePeriod=2 Jul 27 21:51:10.253166 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:10.253138 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k9xnl" event={"ID":"57c2f4c0-758c-4ff9-9e75-2437780c3f62","Type":"ContainerStarted","Data":"94035c43e2c4990bd1b3aff219ca7833bd463280fcbd784dcc6deeb4381d8ec4"} Jul 27 21:51:10.268667 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:10.268609 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-khdc6" podStartSLOduration=1.723327309 podStartE2EDuration="4.268589307s" totalImagesPullingTime="2.545261998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:51:10.26846607 +0000 UTC m=+347.885903402" watchObservedRunningTime="2026-07-27 21:51:10.268589307 +0000 UTC m=+347.886026627" Jul 27 21:51:10.284499 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:10.284438 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-k9xnl" podStartSLOduration=1.316400427 podStartE2EDuration="3.284424778s" totalImagesPullingTime="1.968024351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:51:10.282987906 +0000 UTC m=+347.900425226" watchObservedRunningTime="2026-07-27 21:51:10.284424778 +0000 UTC m=+347.901862097" Jul 27 21:51:10.483752 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:10.483725 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-khdc6" Jul 27 21:51:10.497415 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:10.497392 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ng6w\" (UniqueName: \"kubernetes.io/projected/055b597f-59a9-4d71-b9ef-81c3033a988f-kube-api-access-7ng6w\") pod \"055b597f-59a9-4d71-b9ef-81c3033a988f\" (UID: \"055b597f-59a9-4d71-b9ef-81c3033a988f\") " Jul 27 21:51:10.499461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:10.499437 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055b597f-59a9-4d71-b9ef-81c3033a988f-kube-api-access-7ng6w" (OuterVolumeSpecName: "kube-api-access-7ng6w") pod "055b597f-59a9-4d71-b9ef-81c3033a988f" (UID: "055b597f-59a9-4d71-b9ef-81c3033a988f"). InnerVolumeSpecName "kube-api-access-7ng6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:51:10.598540 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:10.598458 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ng6w\" (UniqueName: \"kubernetes.io/projected/055b597f-59a9-4d71-b9ef-81c3033a988f-kube-api-access-7ng6w\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:51:11.258488 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:11.258455 2561 generic.go:356] "Generic (PLEG): container finished" podID="055b597f-59a9-4d71-b9ef-81c3033a988f" containerID="e3cc17d9211540f3eded1c85aef5a683dd49037f265d0d3ca4e1e9902d3ba170" exitCode=0 Jul 27 21:51:11.258950 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:11.258497 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-khdc6" Jul 27 21:51:11.258950 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:11.258544 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-khdc6" event={"ID":"055b597f-59a9-4d71-b9ef-81c3033a988f","Type":"ContainerDied","Data":"e3cc17d9211540f3eded1c85aef5a683dd49037f265d0d3ca4e1e9902d3ba170"} Jul 27 21:51:11.258950 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:11.258599 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-khdc6" event={"ID":"055b597f-59a9-4d71-b9ef-81c3033a988f","Type":"ContainerDied","Data":"5f0f33e30c787ea194b0248114681641735871bd89d483964b19bc436df09f5c"} Jul 27 21:51:11.258950 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:11.258628 2561 scope.go:117] "RemoveContainer" containerID="e3cc17d9211540f3eded1c85aef5a683dd49037f265d0d3ca4e1e9902d3ba170" Jul 27 21:51:11.267021 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:11.267004 2561 scope.go:117] "RemoveContainer" containerID="e3cc17d9211540f3eded1c85aef5a683dd49037f265d0d3ca4e1e9902d3ba170" Jul 27 21:51:11.267290 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:51:11.267250 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3cc17d9211540f3eded1c85aef5a683dd49037f265d0d3ca4e1e9902d3ba170\": container with ID starting with e3cc17d9211540f3eded1c85aef5a683dd49037f265d0d3ca4e1e9902d3ba170 not found: ID does not exist" containerID="e3cc17d9211540f3eded1c85aef5a683dd49037f265d0d3ca4e1e9902d3ba170" Jul 27 21:51:11.267374 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:11.267298 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3cc17d9211540f3eded1c85aef5a683dd49037f265d0d3ca4e1e9902d3ba170"} err="failed to get container status \"e3cc17d9211540f3eded1c85aef5a683dd49037f265d0d3ca4e1e9902d3ba170\": rpc error: code = NotFound desc = could not find container \"e3cc17d9211540f3eded1c85aef5a683dd49037f265d0d3ca4e1e9902d3ba170\": container with ID starting with e3cc17d9211540f3eded1c85aef5a683dd49037f265d0d3ca4e1e9902d3ba170 not found: ID does not exist" Jul 27 21:51:11.274250 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:11.274227 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-khdc6"] Jul 27 21:51:11.278253 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:11.278230 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-khdc6"] Jul 27 21:51:12.817901 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:12.817871 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="055b597f-59a9-4d71-b9ef-81c3033a988f" path="/var/lib/kubelet/pods/055b597f-59a9-4d71-b9ef-81c3033a988f/volumes" Jul 27 21:51:17.584342 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:17.584202 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-k9xnl" Jul 27 21:51:17.584342 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:17.584311 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-k9xnl" Jul 27 21:51:17.605476 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:17.605451 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-k9xnl" Jul 27 21:51:18.309362 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:18.309337 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-k9xnl" Jul 27 21:51:22.081903 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.081869 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk"] Jul 27 21:51:22.082412 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.082163 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="055b597f-59a9-4d71-b9ef-81c3033a988f" containerName="registry-server" Jul 27 21:51:22.082412 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.082174 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="055b597f-59a9-4d71-b9ef-81c3033a988f" containerName="registry-server" Jul 27 21:51:22.082412 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.082224 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="055b597f-59a9-4d71-b9ef-81c3033a988f" containerName="registry-server" Jul 27 21:51:22.084980 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.084963 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" Jul 27 21:51:22.087145 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.087122 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r79nw\"" Jul 27 21:51:22.093639 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.093618 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk"] Jul 27 21:51:22.186932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.186898 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71695e96-7c86-47b9-a806-cfed10d81d7d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk\" (UID: \"71695e96-7c86-47b9-a806-cfed10d81d7d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" Jul 27 21:51:22.186932 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.186938 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71695e96-7c86-47b9-a806-cfed10d81d7d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk\" (UID: \"71695e96-7c86-47b9-a806-cfed10d81d7d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" Jul 27 21:51:22.187238 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.186972 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsdfl\" (UniqueName: \"kubernetes.io/projected/71695e96-7c86-47b9-a806-cfed10d81d7d-kube-api-access-dsdfl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk\" (UID: \"71695e96-7c86-47b9-a806-cfed10d81d7d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" Jul 27 21:51:22.288405 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.288352 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71695e96-7c86-47b9-a806-cfed10d81d7d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk\" (UID: \"71695e96-7c86-47b9-a806-cfed10d81d7d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" Jul 27 21:51:22.288615 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.288418 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71695e96-7c86-47b9-a806-cfed10d81d7d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk\" (UID: \"71695e96-7c86-47b9-a806-cfed10d81d7d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" Jul 27 21:51:22.288615 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.288472 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsdfl\" (UniqueName: \"kubernetes.io/projected/71695e96-7c86-47b9-a806-cfed10d81d7d-kube-api-access-dsdfl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk\" (UID: \"71695e96-7c86-47b9-a806-cfed10d81d7d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" Jul 27 21:51:22.288794 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.288772 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71695e96-7c86-47b9-a806-cfed10d81d7d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk\" (UID: \"71695e96-7c86-47b9-a806-cfed10d81d7d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" Jul 27 21:51:22.288866 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.288827 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71695e96-7c86-47b9-a806-cfed10d81d7d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk\" (UID: \"71695e96-7c86-47b9-a806-cfed10d81d7d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" Jul 27 21:51:22.296733 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.296708 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsdfl\" (UniqueName: \"kubernetes.io/projected/71695e96-7c86-47b9-a806-cfed10d81d7d-kube-api-access-dsdfl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk\" (UID: \"71695e96-7c86-47b9-a806-cfed10d81d7d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" Jul 27 21:51:22.394573 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.394488 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" Jul 27 21:51:22.544241 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.544218 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk"] Jul 27 21:51:22.546512 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:51:22.546479 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71695e96_7c86_47b9_a806_cfed10d81d7d.slice/crio-7acde3957e3778f81da84eb9e1b2764d2dd73192b48b4e538b2b6d156718c3e2 WatchSource:0}: Error finding container 7acde3957e3778f81da84eb9e1b2764d2dd73192b48b4e538b2b6d156718c3e2: Status 404 returned error can't find the container with id 7acde3957e3778f81da84eb9e1b2764d2dd73192b48b4e538b2b6d156718c3e2 Jul 27 21:51:22.675022 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.674989 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf"] Jul 27 21:51:22.678135 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.678119 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" Jul 27 21:51:22.686139 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.686119 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf"] Jul 27 21:51:22.793760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.793725 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2df13706-8850-4edb-b6b9-9495738b4b95-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf\" (UID: \"2df13706-8850-4edb-b6b9-9495738b4b95\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" Jul 27 21:51:22.793910 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.793816 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2df13706-8850-4edb-b6b9-9495738b4b95-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf\" (UID: \"2df13706-8850-4edb-b6b9-9495738b4b95\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" Jul 27 21:51:22.793910 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.793841 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpp56\" (UniqueName: \"kubernetes.io/projected/2df13706-8850-4edb-b6b9-9495738b4b95-kube-api-access-hpp56\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf\" (UID: \"2df13706-8850-4edb-b6b9-9495738b4b95\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" Jul 27 21:51:22.894357 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.894325 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2df13706-8850-4edb-b6b9-9495738b4b95-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf\" (UID: \"2df13706-8850-4edb-b6b9-9495738b4b95\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" Jul 27 21:51:22.894357 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.894356 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpp56\" (UniqueName: \"kubernetes.io/projected/2df13706-8850-4edb-b6b9-9495738b4b95-kube-api-access-hpp56\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf\" (UID: \"2df13706-8850-4edb-b6b9-9495738b4b95\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" Jul 27 21:51:22.894601 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.894393 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2df13706-8850-4edb-b6b9-9495738b4b95-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf\" (UID: \"2df13706-8850-4edb-b6b9-9495738b4b95\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" Jul 27 21:51:22.894771 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.894747 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2df13706-8850-4edb-b6b9-9495738b4b95-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf\" (UID: \"2df13706-8850-4edb-b6b9-9495738b4b95\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" Jul 27 21:51:22.894771 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.894761 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2df13706-8850-4edb-b6b9-9495738b4b95-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf\" (UID: \"2df13706-8850-4edb-b6b9-9495738b4b95\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" Jul 27 21:51:22.903086 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:22.903065 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpp56\" (UniqueName: \"kubernetes.io/projected/2df13706-8850-4edb-b6b9-9495738b4b95-kube-api-access-hpp56\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf\" (UID: \"2df13706-8850-4edb-b6b9-9495738b4b95\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" Jul 27 21:51:23.000225 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.000145 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" Jul 27 21:51:23.083854 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.083826 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6"] Jul 27 21:51:23.088271 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.088232 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" Jul 27 21:51:23.097032 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.097007 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6"] Jul 27 21:51:23.132085 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.132061 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf"] Jul 27 21:51:23.134231 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:51:23.134198 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df13706_8850_4edb_b6b9_9495738b4b95.slice/crio-3be4feb7cf9a97a5cc6b4d274dbf5f14f05e4d2b39c6c3ef69410a23055b9445 WatchSource:0}: Error finding container 3be4feb7cf9a97a5cc6b4d274dbf5f14f05e4d2b39c6c3ef69410a23055b9445: Status 404 returned error can't find the container with id 3be4feb7cf9a97a5cc6b4d274dbf5f14f05e4d2b39c6c3ef69410a23055b9445 Jul 27 21:51:23.197682 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.197651 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16d3de7a-afc6-46bd-8bbf-2833180ba288-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6\" (UID: \"16d3de7a-afc6-46bd-8bbf-2833180ba288\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" Jul 27 21:51:23.197811 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.197716 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h8w2\" (UniqueName: \"kubernetes.io/projected/16d3de7a-afc6-46bd-8bbf-2833180ba288-kube-api-access-6h8w2\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6\" (UID: \"16d3de7a-afc6-46bd-8bbf-2833180ba288\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" Jul 27 21:51:23.197811 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.197746 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16d3de7a-afc6-46bd-8bbf-2833180ba288-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6\" (UID: \"16d3de7a-afc6-46bd-8bbf-2833180ba288\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" Jul 27 21:51:23.298735 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.298649 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16d3de7a-afc6-46bd-8bbf-2833180ba288-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6\" (UID: \"16d3de7a-afc6-46bd-8bbf-2833180ba288\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" Jul 27 21:51:23.298735 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.298708 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h8w2\" (UniqueName: \"kubernetes.io/projected/16d3de7a-afc6-46bd-8bbf-2833180ba288-kube-api-access-6h8w2\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6\" (UID: \"16d3de7a-afc6-46bd-8bbf-2833180ba288\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" Jul 27 21:51:23.298910 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.298738 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16d3de7a-afc6-46bd-8bbf-2833180ba288-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6\" (UID: \"16d3de7a-afc6-46bd-8bbf-2833180ba288\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" Jul 27 21:51:23.299044 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.299024 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16d3de7a-afc6-46bd-8bbf-2833180ba288-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6\" (UID: \"16d3de7a-afc6-46bd-8bbf-2833180ba288\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" Jul 27 21:51:23.299097 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.299078 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16d3de7a-afc6-46bd-8bbf-2833180ba288-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6\" (UID: \"16d3de7a-afc6-46bd-8bbf-2833180ba288\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" Jul 27 21:51:23.308402 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.308373 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h8w2\" (UniqueName: \"kubernetes.io/projected/16d3de7a-afc6-46bd-8bbf-2833180ba288-kube-api-access-6h8w2\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6\" (UID: \"16d3de7a-afc6-46bd-8bbf-2833180ba288\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" Jul 27 21:51:23.310711 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.310685 2561 generic.go:356] "Generic (PLEG): container finished" podID="71695e96-7c86-47b9-a806-cfed10d81d7d" containerID="efefd9b15da9cfc5494c25f0d1ad3034dd1be5133f69199d0fe389a581063d87" exitCode=0 Jul 27 21:51:23.310838 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.310770 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" event={"ID":"71695e96-7c86-47b9-a806-cfed10d81d7d","Type":"ContainerDied","Data":"efefd9b15da9cfc5494c25f0d1ad3034dd1be5133f69199d0fe389a581063d87"} Jul 27 21:51:23.310893 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.310829 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" event={"ID":"71695e96-7c86-47b9-a806-cfed10d81d7d","Type":"ContainerStarted","Data":"7acde3957e3778f81da84eb9e1b2764d2dd73192b48b4e538b2b6d156718c3e2"} Jul 27 21:51:23.315331 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.312631 2561 generic.go:356] "Generic (PLEG): container finished" podID="2df13706-8850-4edb-b6b9-9495738b4b95" containerID="99e97ff2782f0ee495fabadd2aa8c8cf6887ba77a296ab88fdb6b1baada46882" exitCode=0 Jul 27 21:51:23.315331 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.312707 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" event={"ID":"2df13706-8850-4edb-b6b9-9495738b4b95","Type":"ContainerDied","Data":"99e97ff2782f0ee495fabadd2aa8c8cf6887ba77a296ab88fdb6b1baada46882"} Jul 27 21:51:23.315331 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.312767 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" event={"ID":"2df13706-8850-4edb-b6b9-9495738b4b95","Type":"ContainerStarted","Data":"3be4feb7cf9a97a5cc6b4d274dbf5f14f05e4d2b39c6c3ef69410a23055b9445"} Jul 27 21:51:23.398827 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.398779 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" Jul 27 21:51:23.483580 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.483548 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx"] Jul 27 21:51:23.488714 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.488692 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" Jul 27 21:51:23.494488 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.494461 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx"] Jul 27 21:51:23.540618 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.540587 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6"] Jul 27 21:51:23.542249 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:51:23.542218 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d3de7a_afc6_46bd_8bbf_2833180ba288.slice/crio-ec721ac0af26b4ece40b6f51a1933ca214ce05db48fa749f26e2036bfa68c747 WatchSource:0}: Error finding container ec721ac0af26b4ece40b6f51a1933ca214ce05db48fa749f26e2036bfa68c747: Status 404 returned error can't find the container with id ec721ac0af26b4ece40b6f51a1933ca214ce05db48fa749f26e2036bfa68c747 Jul 27 21:51:23.602382 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.602350 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfrqv\" (UniqueName: \"kubernetes.io/projected/bf5626a4-3553-47be-b521-41df7e2bbd9d-kube-api-access-mfrqv\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx\" (UID: \"bf5626a4-3553-47be-b521-41df7e2bbd9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" Jul 27 21:51:23.602496 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.602460 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5626a4-3553-47be-b521-41df7e2bbd9d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx\" (UID: \"bf5626a4-3553-47be-b521-41df7e2bbd9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" Jul 27 21:51:23.602537 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.602507 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5626a4-3553-47be-b521-41df7e2bbd9d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx\" (UID: \"bf5626a4-3553-47be-b521-41df7e2bbd9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" Jul 27 21:51:23.703406 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.703376 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfrqv\" (UniqueName: \"kubernetes.io/projected/bf5626a4-3553-47be-b521-41df7e2bbd9d-kube-api-access-mfrqv\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx\" (UID: \"bf5626a4-3553-47be-b521-41df7e2bbd9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" Jul 27 21:51:23.703568 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.703434 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5626a4-3553-47be-b521-41df7e2bbd9d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx\" (UID: \"bf5626a4-3553-47be-b521-41df7e2bbd9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" Jul 27 21:51:23.703568 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.703469 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5626a4-3553-47be-b521-41df7e2bbd9d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx\" (UID: \"bf5626a4-3553-47be-b521-41df7e2bbd9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" Jul 27 21:51:23.703852 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.703832 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5626a4-3553-47be-b521-41df7e2bbd9d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx\" (UID: \"bf5626a4-3553-47be-b521-41df7e2bbd9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" Jul 27 21:51:23.703892 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.703876 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5626a4-3553-47be-b521-41df7e2bbd9d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx\" (UID: \"bf5626a4-3553-47be-b521-41df7e2bbd9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" Jul 27 21:51:23.716080 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.716058 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfrqv\" (UniqueName: \"kubernetes.io/projected/bf5626a4-3553-47be-b521-41df7e2bbd9d-kube-api-access-mfrqv\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx\" (UID: \"bf5626a4-3553-47be-b521-41df7e2bbd9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" Jul 27 21:51:23.799762 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.799733 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" Jul 27 21:51:23.926696 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:23.926674 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx"] Jul 27 21:51:23.928844 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:51:23.928822 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5626a4_3553_47be_b521_41df7e2bbd9d.slice/crio-7992afc869a829559c9bdaed9431f17066a7200baaf807ced1508cd29a5769f2 WatchSource:0}: Error finding container 7992afc869a829559c9bdaed9431f17066a7200baaf807ced1508cd29a5769f2: Status 404 returned error can't find the container with id 7992afc869a829559c9bdaed9431f17066a7200baaf807ced1508cd29a5769f2 Jul 27 21:51:24.319552 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:24.319517 2561 generic.go:356] "Generic (PLEG): container finished" podID="71695e96-7c86-47b9-a806-cfed10d81d7d" containerID="c2a9df7310e7cb3fee39090aad5479a1236da9710be776ad8e97b2834665ca59" exitCode=0 Jul 27 21:51:24.319993 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:24.319604 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" event={"ID":"71695e96-7c86-47b9-a806-cfed10d81d7d","Type":"ContainerDied","Data":"c2a9df7310e7cb3fee39090aad5479a1236da9710be776ad8e97b2834665ca59"} Jul 27 21:51:24.321127 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:24.321104 2561 generic.go:356] "Generic (PLEG): container finished" podID="bf5626a4-3553-47be-b521-41df7e2bbd9d" containerID="6ddf775555f367d8796e0715d87d4befd09dc2919e64abe9a9b02f529a8d5a3b" exitCode=0 Jul 27 21:51:24.321232 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:24.321141 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" event={"ID":"bf5626a4-3553-47be-b521-41df7e2bbd9d","Type":"ContainerDied","Data":"6ddf775555f367d8796e0715d87d4befd09dc2919e64abe9a9b02f529a8d5a3b"} Jul 27 21:51:24.321232 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:24.321181 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" event={"ID":"bf5626a4-3553-47be-b521-41df7e2bbd9d","Type":"ContainerStarted","Data":"7992afc869a829559c9bdaed9431f17066a7200baaf807ced1508cd29a5769f2"} Jul 27 21:51:24.322725 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:24.322707 2561 generic.go:356] "Generic (PLEG): container finished" podID="16d3de7a-afc6-46bd-8bbf-2833180ba288" containerID="aa66c80ac21f1e6d1bb0307d0fa082ea9ddba01f67aa8b101124e26cdf13c18b" exitCode=0 Jul 27 21:51:24.322791 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:24.322734 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" event={"ID":"16d3de7a-afc6-46bd-8bbf-2833180ba288","Type":"ContainerDied","Data":"aa66c80ac21f1e6d1bb0307d0fa082ea9ddba01f67aa8b101124e26cdf13c18b"} Jul 27 21:51:24.322791 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:24.322772 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" event={"ID":"16d3de7a-afc6-46bd-8bbf-2833180ba288","Type":"ContainerStarted","Data":"ec721ac0af26b4ece40b6f51a1933ca214ce05db48fa749f26e2036bfa68c747"} Jul 27 21:51:25.329967 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:25.329937 2561 generic.go:356] "Generic (PLEG): container finished" podID="2df13706-8850-4edb-b6b9-9495738b4b95" containerID="17d8a2fa1bde3a030342ae8b283ce9b91ca25b99dae2c40350bda903555d6c7e" exitCode=0 Jul 27 21:51:25.330348 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:25.330024 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" event={"ID":"2df13706-8850-4edb-b6b9-9495738b4b95","Type":"ContainerDied","Data":"17d8a2fa1bde3a030342ae8b283ce9b91ca25b99dae2c40350bda903555d6c7e"} Jul 27 21:51:25.332499 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:25.332473 2561 generic.go:356] "Generic (PLEG): container finished" podID="71695e96-7c86-47b9-a806-cfed10d81d7d" containerID="fb618a02a1d367fa4615b1016d28b272da91b7c2d1a52d0ac580bc3b3c9447aa" exitCode=0 Jul 27 21:51:25.332600 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:25.332554 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" event={"ID":"71695e96-7c86-47b9-a806-cfed10d81d7d","Type":"ContainerDied","Data":"fb618a02a1d367fa4615b1016d28b272da91b7c2d1a52d0ac580bc3b3c9447aa"} Jul 27 21:51:26.339157 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.339118 2561 generic.go:356] "Generic (PLEG): container finished" podID="2df13706-8850-4edb-b6b9-9495738b4b95" containerID="e35e473eda59b5999b5c893d25cbbd4304b9f69152b9f1b382b32958a81a2396" exitCode=0 Jul 27 21:51:26.339556 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.339199 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" event={"ID":"2df13706-8850-4edb-b6b9-9495738b4b95","Type":"ContainerDied","Data":"e35e473eda59b5999b5c893d25cbbd4304b9f69152b9f1b382b32958a81a2396"} Jul 27 21:51:26.340829 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.340801 2561 generic.go:356] "Generic (PLEG): container finished" podID="16d3de7a-afc6-46bd-8bbf-2833180ba288" containerID="c6feba6c64acb0d082bef60e8a7a3d4cd57c64854d9a16b20af419ddd8c031a3" exitCode=0 Jul 27 21:51:26.340918 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.340885 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" event={"ID":"16d3de7a-afc6-46bd-8bbf-2833180ba288","Type":"ContainerDied","Data":"c6feba6c64acb0d082bef60e8a7a3d4cd57c64854d9a16b20af419ddd8c031a3"} Jul 27 21:51:26.342637 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.342617 2561 generic.go:356] "Generic (PLEG): container finished" podID="bf5626a4-3553-47be-b521-41df7e2bbd9d" containerID="40dcca0f8450165b9205bf10446e24c7494009573a2de137bd775ead751bef79" exitCode=0 Jul 27 21:51:26.342731 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.342696 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" event={"ID":"bf5626a4-3553-47be-b521-41df7e2bbd9d","Type":"ContainerDied","Data":"40dcca0f8450165b9205bf10446e24c7494009573a2de137bd775ead751bef79"} Jul 27 21:51:26.488654 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.488625 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" Jul 27 21:51:26.525517 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.525494 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsdfl\" (UniqueName: \"kubernetes.io/projected/71695e96-7c86-47b9-a806-cfed10d81d7d-kube-api-access-dsdfl\") pod \"71695e96-7c86-47b9-a806-cfed10d81d7d\" (UID: \"71695e96-7c86-47b9-a806-cfed10d81d7d\") " Jul 27 21:51:26.525641 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.525561 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71695e96-7c86-47b9-a806-cfed10d81d7d-bundle\") pod \"71695e96-7c86-47b9-a806-cfed10d81d7d\" (UID: \"71695e96-7c86-47b9-a806-cfed10d81d7d\") " Jul 27 21:51:26.525641 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.525604 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71695e96-7c86-47b9-a806-cfed10d81d7d-util\") pod \"71695e96-7c86-47b9-a806-cfed10d81d7d\" (UID: \"71695e96-7c86-47b9-a806-cfed10d81d7d\") " Jul 27 21:51:26.526102 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.526072 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71695e96-7c86-47b9-a806-cfed10d81d7d-bundle" (OuterVolumeSpecName: "bundle") pod "71695e96-7c86-47b9-a806-cfed10d81d7d" (UID: "71695e96-7c86-47b9-a806-cfed10d81d7d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:51:26.527842 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.527813 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71695e96-7c86-47b9-a806-cfed10d81d7d-kube-api-access-dsdfl" (OuterVolumeSpecName: "kube-api-access-dsdfl") pod "71695e96-7c86-47b9-a806-cfed10d81d7d" (UID: "71695e96-7c86-47b9-a806-cfed10d81d7d"). InnerVolumeSpecName "kube-api-access-dsdfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:51:26.535123 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.535071 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71695e96-7c86-47b9-a806-cfed10d81d7d-util" (OuterVolumeSpecName: "util") pod "71695e96-7c86-47b9-a806-cfed10d81d7d" (UID: "71695e96-7c86-47b9-a806-cfed10d81d7d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:51:26.626209 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.626119 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsdfl\" (UniqueName: \"kubernetes.io/projected/71695e96-7c86-47b9-a806-cfed10d81d7d-kube-api-access-dsdfl\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:51:26.626209 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.626150 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71695e96-7c86-47b9-a806-cfed10d81d7d-bundle\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:51:26.626209 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:26.626161 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71695e96-7c86-47b9-a806-cfed10d81d7d-util\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:51:27.348779 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.348746 2561 generic.go:356] "Generic (PLEG): container finished" podID="bf5626a4-3553-47be-b521-41df7e2bbd9d" containerID="48cb55c287303e733e03cec675e26046fe7bdfc6a0a38dcc01e29607a8775d05" exitCode=0 Jul 27 21:51:27.349169 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.348860 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" event={"ID":"bf5626a4-3553-47be-b521-41df7e2bbd9d","Type":"ContainerDied","Data":"48cb55c287303e733e03cec675e26046fe7bdfc6a0a38dcc01e29607a8775d05"} Jul 27 21:51:27.350749 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.350727 2561 generic.go:356] "Generic (PLEG): container finished" podID="16d3de7a-afc6-46bd-8bbf-2833180ba288" containerID="320913fbe89c6f17e50b4f3e47b3248b25bd0b85acff73056687defb529fb3ec" exitCode=0 Jul 27 21:51:27.350837 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.350774 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" event={"ID":"16d3de7a-afc6-46bd-8bbf-2833180ba288","Type":"ContainerDied","Data":"320913fbe89c6f17e50b4f3e47b3248b25bd0b85acff73056687defb529fb3ec"} Jul 27 21:51:27.352507 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.352476 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" event={"ID":"71695e96-7c86-47b9-a806-cfed10d81d7d","Type":"ContainerDied","Data":"7acde3957e3778f81da84eb9e1b2764d2dd73192b48b4e538b2b6d156718c3e2"} Jul 27 21:51:27.352592 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.352522 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk" Jul 27 21:51:27.352592 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.352523 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7acde3957e3778f81da84eb9e1b2764d2dd73192b48b4e538b2b6d156718c3e2" Jul 27 21:51:27.478516 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.478491 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" Jul 27 21:51:27.533710 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.533677 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpp56\" (UniqueName: \"kubernetes.io/projected/2df13706-8850-4edb-b6b9-9495738b4b95-kube-api-access-hpp56\") pod \"2df13706-8850-4edb-b6b9-9495738b4b95\" (UID: \"2df13706-8850-4edb-b6b9-9495738b4b95\") " Jul 27 21:51:27.533876 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.533758 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2df13706-8850-4edb-b6b9-9495738b4b95-util\") pod \"2df13706-8850-4edb-b6b9-9495738b4b95\" (UID: \"2df13706-8850-4edb-b6b9-9495738b4b95\") " Jul 27 21:51:27.533935 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.533867 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2df13706-8850-4edb-b6b9-9495738b4b95-bundle\") pod \"2df13706-8850-4edb-b6b9-9495738b4b95\" (UID: \"2df13706-8850-4edb-b6b9-9495738b4b95\") " Jul 27 21:51:27.534452 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.534430 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df13706-8850-4edb-b6b9-9495738b4b95-bundle" (OuterVolumeSpecName: "bundle") pod "2df13706-8850-4edb-b6b9-9495738b4b95" (UID: "2df13706-8850-4edb-b6b9-9495738b4b95"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:51:27.535755 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.535737 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df13706-8850-4edb-b6b9-9495738b4b95-kube-api-access-hpp56" (OuterVolumeSpecName: "kube-api-access-hpp56") pod "2df13706-8850-4edb-b6b9-9495738b4b95" (UID: "2df13706-8850-4edb-b6b9-9495738b4b95"). InnerVolumeSpecName "kube-api-access-hpp56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:51:27.538905 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.538882 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df13706-8850-4edb-b6b9-9495738b4b95-util" (OuterVolumeSpecName: "util") pod "2df13706-8850-4edb-b6b9-9495738b4b95" (UID: "2df13706-8850-4edb-b6b9-9495738b4b95"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:51:27.635494 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.635411 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpp56\" (UniqueName: \"kubernetes.io/projected/2df13706-8850-4edb-b6b9-9495738b4b95-kube-api-access-hpp56\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:51:27.635494 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.635438 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2df13706-8850-4edb-b6b9-9495738b4b95-util\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:51:27.635494 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:27.635447 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2df13706-8850-4edb-b6b9-9495738b4b95-bundle\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:51:28.358576 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.358537 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" event={"ID":"2df13706-8850-4edb-b6b9-9495738b4b95","Type":"ContainerDied","Data":"3be4feb7cf9a97a5cc6b4d274dbf5f14f05e4d2b39c6c3ef69410a23055b9445"} Jul 27 21:51:28.358576 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.358573 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf" Jul 27 21:51:28.359034 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.358598 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3be4feb7cf9a97a5cc6b4d274dbf5f14f05e4d2b39c6c3ef69410a23055b9445" Jul 27 21:51:28.506637 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.506611 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" Jul 27 21:51:28.508709 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.508692 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" Jul 27 21:51:28.543763 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.543721 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h8w2\" (UniqueName: \"kubernetes.io/projected/16d3de7a-afc6-46bd-8bbf-2833180ba288-kube-api-access-6h8w2\") pod \"16d3de7a-afc6-46bd-8bbf-2833180ba288\" (UID: \"16d3de7a-afc6-46bd-8bbf-2833180ba288\") " Jul 27 21:51:28.544013 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.543809 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16d3de7a-afc6-46bd-8bbf-2833180ba288-util\") pod \"16d3de7a-afc6-46bd-8bbf-2833180ba288\" (UID: \"16d3de7a-afc6-46bd-8bbf-2833180ba288\") " Jul 27 21:51:28.544013 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.543844 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfrqv\" (UniqueName: \"kubernetes.io/projected/bf5626a4-3553-47be-b521-41df7e2bbd9d-kube-api-access-mfrqv\") pod \"bf5626a4-3553-47be-b521-41df7e2bbd9d\" (UID: \"bf5626a4-3553-47be-b521-41df7e2bbd9d\") " Jul 27 21:51:28.544013 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.543883 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5626a4-3553-47be-b521-41df7e2bbd9d-util\") pod \"bf5626a4-3553-47be-b521-41df7e2bbd9d\" (UID: \"bf5626a4-3553-47be-b521-41df7e2bbd9d\") " Jul 27 21:51:28.544013 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.543909 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16d3de7a-afc6-46bd-8bbf-2833180ba288-bundle\") pod \"16d3de7a-afc6-46bd-8bbf-2833180ba288\" (UID: \"16d3de7a-afc6-46bd-8bbf-2833180ba288\") " Jul 27 21:51:28.544013 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.543932 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5626a4-3553-47be-b521-41df7e2bbd9d-bundle\") pod \"bf5626a4-3553-47be-b521-41df7e2bbd9d\" (UID: \"bf5626a4-3553-47be-b521-41df7e2bbd9d\") " Jul 27 21:51:28.544650 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.544563 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d3de7a-afc6-46bd-8bbf-2833180ba288-bundle" (OuterVolumeSpecName: "bundle") pod "16d3de7a-afc6-46bd-8bbf-2833180ba288" (UID: "16d3de7a-afc6-46bd-8bbf-2833180ba288"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:51:28.544758 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.544669 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5626a4-3553-47be-b521-41df7e2bbd9d-bundle" (OuterVolumeSpecName: "bundle") pod "bf5626a4-3553-47be-b521-41df7e2bbd9d" (UID: "bf5626a4-3553-47be-b521-41df7e2bbd9d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:51:28.547734 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.547704 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d3de7a-afc6-46bd-8bbf-2833180ba288-kube-api-access-6h8w2" (OuterVolumeSpecName: "kube-api-access-6h8w2") pod "16d3de7a-afc6-46bd-8bbf-2833180ba288" (UID: "16d3de7a-afc6-46bd-8bbf-2833180ba288"). InnerVolumeSpecName "kube-api-access-6h8w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:51:28.548479 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.548453 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5626a4-3553-47be-b521-41df7e2bbd9d-kube-api-access-mfrqv" (OuterVolumeSpecName: "kube-api-access-mfrqv") pod "bf5626a4-3553-47be-b521-41df7e2bbd9d" (UID: "bf5626a4-3553-47be-b521-41df7e2bbd9d"). InnerVolumeSpecName "kube-api-access-mfrqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:51:28.550564 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.550540 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5626a4-3553-47be-b521-41df7e2bbd9d-util" (OuterVolumeSpecName: "util") pod "bf5626a4-3553-47be-b521-41df7e2bbd9d" (UID: "bf5626a4-3553-47be-b521-41df7e2bbd9d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:51:28.551166 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.551141 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d3de7a-afc6-46bd-8bbf-2833180ba288-util" (OuterVolumeSpecName: "util") pod "16d3de7a-afc6-46bd-8bbf-2833180ba288" (UID: "16d3de7a-afc6-46bd-8bbf-2833180ba288"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:51:28.645384 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.645302 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16d3de7a-afc6-46bd-8bbf-2833180ba288-util\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:51:28.645384 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.645330 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfrqv\" (UniqueName: \"kubernetes.io/projected/bf5626a4-3553-47be-b521-41df7e2bbd9d-kube-api-access-mfrqv\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:51:28.645384 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.645343 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5626a4-3553-47be-b521-41df7e2bbd9d-util\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:51:28.645384 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.645351 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16d3de7a-afc6-46bd-8bbf-2833180ba288-bundle\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:51:28.645384 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.645359 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5626a4-3553-47be-b521-41df7e2bbd9d-bundle\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:51:28.645384 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:28.645367 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h8w2\" (UniqueName: \"kubernetes.io/projected/16d3de7a-afc6-46bd-8bbf-2833180ba288-kube-api-access-6h8w2\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:51:29.364686 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:29.364657 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" Jul 27 21:51:29.365087 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:29.364687 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx" event={"ID":"bf5626a4-3553-47be-b521-41df7e2bbd9d","Type":"ContainerDied","Data":"7992afc869a829559c9bdaed9431f17066a7200baaf807ced1508cd29a5769f2"} Jul 27 21:51:29.365087 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:29.364744 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7992afc869a829559c9bdaed9431f17066a7200baaf807ced1508cd29a5769f2" Jul 27 21:51:29.366666 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:29.366592 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" event={"ID":"16d3de7a-afc6-46bd-8bbf-2833180ba288","Type":"ContainerDied","Data":"ec721ac0af26b4ece40b6f51a1933ca214ce05db48fa749f26e2036bfa68c747"} Jul 27 21:51:29.366666 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:29.366641 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec721ac0af26b4ece40b6f51a1933ca214ce05db48fa749f26e2036bfa68c747" Jul 27 21:51:29.366666 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:29.366647 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6" Jul 27 21:51:35.531804 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:35.531770 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:38.177234 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177191 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg"] Jul 27 21:51:38.177760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177646 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df13706-8850-4edb-b6b9-9495738b4b95" containerName="pull" Jul 27 21:51:38.177760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177663 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df13706-8850-4edb-b6b9-9495738b4b95" containerName="pull" Jul 27 21:51:38.177760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177672 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df13706-8850-4edb-b6b9-9495738b4b95" containerName="extract" Jul 27 21:51:38.177760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177680 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df13706-8850-4edb-b6b9-9495738b4b95" containerName="extract" Jul 27 21:51:38.177760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177696 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71695e96-7c86-47b9-a806-cfed10d81d7d" containerName="pull" Jul 27 21:51:38.177760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177705 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="71695e96-7c86-47b9-a806-cfed10d81d7d" containerName="pull" Jul 27 21:51:38.177760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177713 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d3de7a-afc6-46bd-8bbf-2833180ba288" containerName="extract" Jul 27 21:51:38.177760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177720 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d3de7a-afc6-46bd-8bbf-2833180ba288" containerName="extract" Jul 27 21:51:38.177760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177733 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df13706-8850-4edb-b6b9-9495738b4b95" containerName="util" Jul 27 21:51:38.177760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177740 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df13706-8850-4edb-b6b9-9495738b4b95" containerName="util" Jul 27 21:51:38.177760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177747 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5626a4-3553-47be-b521-41df7e2bbd9d" containerName="pull" Jul 27 21:51:38.177760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177754 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5626a4-3553-47be-b521-41df7e2bbd9d" containerName="pull" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177768 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5626a4-3553-47be-b521-41df7e2bbd9d" containerName="util" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177778 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5626a4-3553-47be-b521-41df7e2bbd9d" containerName="util" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177791 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d3de7a-afc6-46bd-8bbf-2833180ba288" containerName="util" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177799 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d3de7a-afc6-46bd-8bbf-2833180ba288" containerName="util" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177807 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71695e96-7c86-47b9-a806-cfed10d81d7d" containerName="extract" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177815 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="71695e96-7c86-47b9-a806-cfed10d81d7d" containerName="extract" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177825 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d3de7a-afc6-46bd-8bbf-2833180ba288" containerName="pull" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177832 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d3de7a-afc6-46bd-8bbf-2833180ba288" containerName="pull" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177842 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5626a4-3553-47be-b521-41df7e2bbd9d" containerName="extract" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177850 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5626a4-3553-47be-b521-41df7e2bbd9d" containerName="extract" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177860 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71695e96-7c86-47b9-a806-cfed10d81d7d" containerName="util" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177868 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="71695e96-7c86-47b9-a806-cfed10d81d7d" containerName="util" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177949 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="71695e96-7c86-47b9-a806-cfed10d81d7d" containerName="extract" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177960 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="16d3de7a-afc6-46bd-8bbf-2833180ba288" containerName="extract" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177968 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf5626a4-3553-47be-b521-41df7e2bbd9d" containerName="extract" Jul 27 21:51:38.178323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.177976 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df13706-8850-4edb-b6b9-9495738b4b95" containerName="extract" Jul 27 21:51:38.182468 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.182448 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:51:38.186037 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.186019 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hsdt9\"" Jul 27 21:51:38.193570 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.193547 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg"] Jul 27 21:51:38.319813 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.319777 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7mw\" (UniqueName: \"kubernetes.io/projected/a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5-kube-api-access-mq7mw\") pod \"limitador-operator-controller-manager-859d7b78bd-kwjdg\" (UID: \"a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:51:38.420710 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.420675 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7mw\" (UniqueName: \"kubernetes.io/projected/a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5-kube-api-access-mq7mw\") pod \"limitador-operator-controller-manager-859d7b78bd-kwjdg\" (UID: \"a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:51:38.429376 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.429321 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7mw\" (UniqueName: \"kubernetes.io/projected/a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5-kube-api-access-mq7mw\") pod \"limitador-operator-controller-manager-859d7b78bd-kwjdg\" (UID: \"a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:51:38.492315 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.492282 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:51:38.624467 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.624443 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg"] Jul 27 21:51:38.626921 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:51:38.626892 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda217fe4f_efb8_4cd9_82f3_6a3f6c1f9ab5.slice/crio-5cacd77a4ab6d1f38209592e3dcc59bb45cf210064ff57ffe9a3acdd3835087f WatchSource:0}: Error finding container 5cacd77a4ab6d1f38209592e3dcc59bb45cf210064ff57ffe9a3acdd3835087f: Status 404 returned error can't find the container with id 5cacd77a4ab6d1f38209592e3dcc59bb45cf210064ff57ffe9a3acdd3835087f Jul 27 21:51:38.926920 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:38.926886 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:39.413640 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:39.413606 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" event={"ID":"a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5","Type":"ContainerStarted","Data":"5cacd77a4ab6d1f38209592e3dcc59bb45cf210064ff57ffe9a3acdd3835087f"} Jul 27 21:51:39.739487 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:39.739407 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:40.422319 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:40.422289 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" event={"ID":"a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5","Type":"ContainerStarted","Data":"fb283a4e1d47ec41d90b36c0e3470b51789d2aebb11e4ccc3e5e1af0b0ae5228"} Jul 27 21:51:40.422740 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:40.422422 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:51:40.439450 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:40.439402 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" podStartSLOduration=0.738031837 podStartE2EDuration="2.439386425s" totalImagesPullingTime="1.701354588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:51:40.437184657 +0000 UTC m=+378.054621974" watchObservedRunningTime="2026-07-27 21:51:40.439386425 +0000 UTC m=+378.056823741" Jul 27 21:51:41.494201 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:41.494169 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:42.214782 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:42.214752 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6kwj2"] Jul 27 21:51:42.217868 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:42.217853 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6kwj2" Jul 27 21:51:42.225876 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:42.225859 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hgxck\"" Jul 27 21:51:42.315729 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:42.315692 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6kwj2"] Jul 27 21:51:42.353667 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:42.353630 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxcwm\" (UniqueName: \"kubernetes.io/projected/2af4f0ad-089c-42d9-9a9d-5f8225eddb3b-kube-api-access-vxcwm\") pod \"authorino-operator-75cc8d5cb7-6kwj2\" (UID: \"2af4f0ad-089c-42d9-9a9d-5f8225eddb3b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6kwj2" Jul 27 21:51:42.454431 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:42.454401 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxcwm\" (UniqueName: \"kubernetes.io/projected/2af4f0ad-089c-42d9-9a9d-5f8225eddb3b-kube-api-access-vxcwm\") pod \"authorino-operator-75cc8d5cb7-6kwj2\" (UID: \"2af4f0ad-089c-42d9-9a9d-5f8225eddb3b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6kwj2" Jul 27 21:51:42.462842 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:42.462820 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxcwm\" (UniqueName: \"kubernetes.io/projected/2af4f0ad-089c-42d9-9a9d-5f8225eddb3b-kube-api-access-vxcwm\") pod \"authorino-operator-75cc8d5cb7-6kwj2\" (UID: \"2af4f0ad-089c-42d9-9a9d-5f8225eddb3b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6kwj2" Jul 27 21:51:42.527020 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:42.526940 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6kwj2" Jul 27 21:51:42.656633 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:42.656512 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6kwj2"] Jul 27 21:51:42.659062 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:51:42.659039 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af4f0ad_089c_42d9_9a9d_5f8225eddb3b.slice/crio-43c8f26fddc1e5c5db1d93d4f1e8fc4f975b5216c0953cb9f8159678651f69d8 WatchSource:0}: Error finding container 43c8f26fddc1e5c5db1d93d4f1e8fc4f975b5216c0953cb9f8159678651f69d8: Status 404 returned error can't find the container with id 43c8f26fddc1e5c5db1d93d4f1e8fc4f975b5216c0953cb9f8159678651f69d8 Jul 27 21:51:42.981615 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:42.981579 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:43.370396 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:43.370318 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:43.437245 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:43.437211 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6kwj2" event={"ID":"2af4f0ad-089c-42d9-9a9d-5f8225eddb3b","Type":"ContainerStarted","Data":"43c8f26fddc1e5c5db1d93d4f1e8fc4f975b5216c0953cb9f8159678651f69d8"} Jul 27 21:51:43.469250 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:43.469219 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:44.973557 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:44.973523 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:45.452794 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:45.452750 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6kwj2" event={"ID":"2af4f0ad-089c-42d9-9a9d-5f8225eddb3b","Type":"ContainerStarted","Data":"202d7813a87a3a87075b082db22c4125a8bf0d47593531724bed081c756fbad4"} Jul 27 21:51:45.452999 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:45.452842 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6kwj2" Jul 27 21:51:45.476206 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:45.476154 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6kwj2" podStartSLOduration=0.951176138 podStartE2EDuration="3.476140089s" totalImagesPullingTime="2.524963951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:51:45.474448575 +0000 UTC m=+383.091885892" watchObservedRunningTime="2026-07-27 21:51:45.476140089 +0000 UTC m=+383.093577407" Jul 27 21:51:46.130300 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:46.130246 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:46.260438 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:46.260401 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:47.998108 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:47.998076 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:51.155436 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:51.155387 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:51.254461 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:51.254432 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:51.429454 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:51.429377 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:51:53.783080 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:53.783035 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:53.884369 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:53.884338 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:55.919175 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:55.919139 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:55.982841 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:55.982811 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:56.460662 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:56.460632 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6kwj2" Jul 27 21:51:56.545155 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:56.545125 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:56.649614 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:56.649582 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:56.984710 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:56.984672 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:57.086047 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:57.085998 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:59.355014 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.354976 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4"] Jul 27 21:51:59.358717 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.358690 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:59.359656 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.359640 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" Jul 27 21:51:59.361587 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.361569 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 27 21:51:59.361927 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.361914 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 27 21:51:59.361982 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.361918 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r79nw\"" Jul 27 21:51:59.365291 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.365245 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4"] Jul 27 21:51:59.495315 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.495282 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzmv9\" (UniqueName: \"kubernetes.io/projected/de01ca40-aac1-44a2-b1a9-e8bafd04dd46-kube-api-access-dzmv9\") pod \"kuadrant-console-plugin-66b5774985-hb6h4\" (UID: \"de01ca40-aac1-44a2-b1a9-e8bafd04dd46\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" Jul 27 21:51:59.495514 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.495358 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/de01ca40-aac1-44a2-b1a9-e8bafd04dd46-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hb6h4\" (UID: \"de01ca40-aac1-44a2-b1a9-e8bafd04dd46\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" Jul 27 21:51:59.495514 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.495410 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/de01ca40-aac1-44a2-b1a9-e8bafd04dd46-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hb6h4\" (UID: \"de01ca40-aac1-44a2-b1a9-e8bafd04dd46\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" Jul 27 21:51:59.495783 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.495762 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:51:59.596110 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.596075 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzmv9\" (UniqueName: \"kubernetes.io/projected/de01ca40-aac1-44a2-b1a9-e8bafd04dd46-kube-api-access-dzmv9\") pod \"kuadrant-console-plugin-66b5774985-hb6h4\" (UID: \"de01ca40-aac1-44a2-b1a9-e8bafd04dd46\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" Jul 27 21:51:59.596301 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.596121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/de01ca40-aac1-44a2-b1a9-e8bafd04dd46-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hb6h4\" (UID: \"de01ca40-aac1-44a2-b1a9-e8bafd04dd46\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" Jul 27 21:51:59.596301 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.596157 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/de01ca40-aac1-44a2-b1a9-e8bafd04dd46-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hb6h4\" (UID: \"de01ca40-aac1-44a2-b1a9-e8bafd04dd46\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" Jul 27 21:51:59.596903 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.596878 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/de01ca40-aac1-44a2-b1a9-e8bafd04dd46-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hb6h4\" (UID: \"de01ca40-aac1-44a2-b1a9-e8bafd04dd46\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" Jul 27 21:51:59.598703 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.598677 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/de01ca40-aac1-44a2-b1a9-e8bafd04dd46-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hb6h4\" (UID: \"de01ca40-aac1-44a2-b1a9-e8bafd04dd46\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" Jul 27 21:51:59.603854 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.603833 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzmv9\" (UniqueName: \"kubernetes.io/projected/de01ca40-aac1-44a2-b1a9-e8bafd04dd46-kube-api-access-dzmv9\") pod \"kuadrant-console-plugin-66b5774985-hb6h4\" (UID: \"de01ca40-aac1-44a2-b1a9-e8bafd04dd46\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" Jul 27 21:51:59.669886 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.669851 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" Jul 27 21:51:59.795886 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.795856 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4"] Jul 27 21:51:59.797855 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:51:59.797829 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde01ca40_aac1_44a2_b1a9_e8bafd04dd46.slice/crio-0842c0615d5f27a759a0da072c0a0572a492199ff82e1a398ded6ab73a1ce23a WatchSource:0}: Error finding container 0842c0615d5f27a759a0da072c0a0572a492199ff82e1a398ded6ab73a1ce23a: Status 404 returned error can't find the container with id 0842c0615d5f27a759a0da072c0a0572a492199ff82e1a398ded6ab73a1ce23a Jul 27 21:51:59.977569 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:51:59.977483 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:00.076242 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:00.076209 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:00.523217 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:00.523181 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" event={"ID":"de01ca40-aac1-44a2-b1a9-e8bafd04dd46","Type":"ContainerStarted","Data":"0842c0615d5f27a759a0da072c0a0572a492199ff82e1a398ded6ab73a1ce23a"} Jul 27 21:52:04.090554 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:04.090434 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:04.189426 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:04.189351 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:07.122397 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:07.122363 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:07.222408 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:07.222374 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:07.281478 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:07.281447 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:09.712449 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:09.712414 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg"] Jul 27 21:52:09.713009 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:09.712780 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" podUID="a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5" containerName="manager" containerID="cri-o://fb283a4e1d47ec41d90b36c0e3470b51789d2aebb11e4ccc3e5e1af0b0ae5228" gracePeriod=2 Jul 27 21:52:09.720536 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:09.720312 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg"] Jul 27 21:52:09.741358 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:09.741327 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9jvjd"] Jul 27 21:52:09.741744 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:09.741724 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5" containerName="manager" Jul 27 21:52:09.741744 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:09.741744 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5" containerName="manager" Jul 27 21:52:09.741934 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:09.741848 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5" containerName="manager" Jul 27 21:52:09.763379 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:09.763344 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9jvjd"] Jul 27 21:52:09.763533 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:09.763468 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9jvjd" Jul 27 21:52:09.765939 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:52:09.765905 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kwjdg\" is forbidden: User \"system:node:ip-10-0-129-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-80.ec2.internal' and this object" podUID="a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:52:09.886585 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:09.886551 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlmwv\" (UniqueName: \"kubernetes.io/projected/453577e3-d603-4333-b7eb-53d08b661966-kube-api-access-vlmwv\") pod \"limitador-operator-controller-manager-859d7b78bd-9jvjd\" (UID: \"453577e3-d603-4333-b7eb-53d08b661966\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9jvjd" Jul 27 21:52:09.987660 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:09.987578 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlmwv\" (UniqueName: \"kubernetes.io/projected/453577e3-d603-4333-b7eb-53d08b661966-kube-api-access-vlmwv\") pod \"limitador-operator-controller-manager-859d7b78bd-9jvjd\" (UID: \"453577e3-d603-4333-b7eb-53d08b661966\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9jvjd" Jul 27 21:52:09.997317 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:09.997282 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlmwv\" (UniqueName: \"kubernetes.io/projected/453577e3-d603-4333-b7eb-53d08b661966-kube-api-access-vlmwv\") pod \"limitador-operator-controller-manager-859d7b78bd-9jvjd\" (UID: \"453577e3-d603-4333-b7eb-53d08b661966\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9jvjd" Jul 27 21:52:10.123148 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:10.122669 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9jvjd" Jul 27 21:52:10.743320 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:10.743245 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:10.853127 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:10.853093 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:14.520638 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:14.520608 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:14.575832 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:14.575794 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:14.714956 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:14.714923 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:14.816553 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:14.816435 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:18.511566 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:18.511535 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:18.566074 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:18.566041 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:21.372401 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.372374 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9jvjd"] Jul 27 21:52:21.372775 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.372423 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:52:21.374314 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:52:21.374288 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kwjdg\" is forbidden: User \"system:node:ip-10-0-129-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-80.ec2.internal' and this object" podUID="a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:52:21.375557 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:52:21.375535 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453577e3_d603_4333_b7eb_53d08b661966.slice/crio-a31c990042d2e4c213a13e386e1d69b33cc4028b6d93fa824d82468d61929cfb WatchSource:0}: Error finding container a31c990042d2e4c213a13e386e1d69b33cc4028b6d93fa824d82468d61929cfb: Status 404 returned error can't find the container with id a31c990042d2e4c213a13e386e1d69b33cc4028b6d93fa824d82468d61929cfb Jul 27 21:52:21.387330 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.387304 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq7mw\" (UniqueName: \"kubernetes.io/projected/a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5-kube-api-access-mq7mw\") pod \"a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5\" (UID: \"a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5\") " Jul 27 21:52:21.389518 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.389492 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5-kube-api-access-mq7mw" (OuterVolumeSpecName: "kube-api-access-mq7mw") pod "a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5" (UID: "a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5"). InnerVolumeSpecName "kube-api-access-mq7mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:52:21.488201 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.488107 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq7mw\" (UniqueName: \"kubernetes.io/projected/a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5-kube-api-access-mq7mw\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:52:21.642132 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.642096 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9jvjd" event={"ID":"453577e3-d603-4333-b7eb-53d08b661966","Type":"ContainerStarted","Data":"2f464728e54a8081460ee724beb6f995926771f4cb8303048f37efa449ebe80b"} Jul 27 21:52:21.642371 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.642154 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9jvjd" event={"ID":"453577e3-d603-4333-b7eb-53d08b661966","Type":"ContainerStarted","Data":"a31c990042d2e4c213a13e386e1d69b33cc4028b6d93fa824d82468d61929cfb"} Jul 27 21:52:21.642371 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.642180 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9jvjd" Jul 27 21:52:21.643616 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.643586 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" event={"ID":"de01ca40-aac1-44a2-b1a9-e8bafd04dd46","Type":"ContainerStarted","Data":"2aa253a23956df29c1d247381aaa08cc36ad657481baa5ac23a3adad06f98400"} Jul 27 21:52:21.643937 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:52:21.643911 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kwjdg\" is forbidden: User \"system:node:ip-10-0-129-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-80.ec2.internal' and this object" podUID="a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:52:21.644859 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.644836 2561 generic.go:356] "Generic (PLEG): container finished" podID="a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5" containerID="fb283a4e1d47ec41d90b36c0e3470b51789d2aebb11e4ccc3e5e1af0b0ae5228" exitCode=0 Jul 27 21:52:21.644949 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.644889 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:52:21.644949 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.644928 2561 scope.go:117] "RemoveContainer" containerID="fb283a4e1d47ec41d90b36c0e3470b51789d2aebb11e4ccc3e5e1af0b0ae5228" Jul 27 21:52:21.653585 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.653564 2561 scope.go:117] "RemoveContainer" containerID="fb283a4e1d47ec41d90b36c0e3470b51789d2aebb11e4ccc3e5e1af0b0ae5228" Jul 27 21:52:21.653862 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:52:21.653843 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb283a4e1d47ec41d90b36c0e3470b51789d2aebb11e4ccc3e5e1af0b0ae5228\": container with ID starting with fb283a4e1d47ec41d90b36c0e3470b51789d2aebb11e4ccc3e5e1af0b0ae5228 not found: ID does not exist" containerID="fb283a4e1d47ec41d90b36c0e3470b51789d2aebb11e4ccc3e5e1af0b0ae5228" Jul 27 21:52:21.653930 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.653874 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb283a4e1d47ec41d90b36c0e3470b51789d2aebb11e4ccc3e5e1af0b0ae5228"} err="failed to get container status \"fb283a4e1d47ec41d90b36c0e3470b51789d2aebb11e4ccc3e5e1af0b0ae5228\": rpc error: code = NotFound desc = could not find container \"fb283a4e1d47ec41d90b36c0e3470b51789d2aebb11e4ccc3e5e1af0b0ae5228\": container with ID starting with fb283a4e1d47ec41d90b36c0e3470b51789d2aebb11e4ccc3e5e1af0b0ae5228 not found: ID does not exist" Jul 27 21:52:21.659617 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.659575 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9jvjd" podStartSLOduration=12.659561534 podStartE2EDuration="12.659561534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:52:21.658505466 +0000 UTC m=+419.275942775" watchObservedRunningTime="2026-07-27 21:52:21.659561534 +0000 UTC m=+419.276998852" Jul 27 21:52:21.673128 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:21.673078 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hb6h4" podStartSLOduration=1.154494567 podStartE2EDuration="22.673062215s" totalImagesPullingTime="21.518567648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:52:21.672191308 +0000 UTC m=+419.289628618" watchObservedRunningTime="2026-07-27 21:52:21.673062215 +0000 UTC m=+419.290499534" Jul 27 21:52:21.673918 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:52:21.673889 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kwjdg\" is forbidden: User \"system:node:ip-10-0-129-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-80.ec2.internal' and this object" podUID="a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:52:22.816817 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:52:22.816774 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kwjdg\" is forbidden: User \"system:node:ip-10-0-129-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-80.ec2.internal' and this object" podUID="a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kwjdg" Jul 27 21:52:22.818312 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:22.818283 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5" path="/var/lib/kubelet/pods/a217fe4f-efb8-4cd9-82f3-6a3f6c1f9ab5/volumes" Jul 27 21:52:23.921613 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:23.921576 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:23.985796 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:23.985766 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:32.652922 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:32.652888 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9jvjd" Jul 27 21:52:42.087090 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.087054 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb"] Jul 27 21:52:42.108820 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.108791 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb"] Jul 27 21:52:42.108974 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.108940 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.111036 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.111017 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sqf6r\"" Jul 27 21:52:42.127513 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.127486 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:42.169795 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.169755 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.169953 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.169797 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.169953 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.169824 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv779\" (UniqueName: \"kubernetes.io/projected/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-kube-api-access-lv779\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.169953 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.169898 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.169953 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.169929 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.170100 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.169960 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.170100 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.169991 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.170100 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.170016 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.170100 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.170031 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.270599 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.270565 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.270599 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.270602 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.270825 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.270733 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv779\" (UniqueName: \"kubernetes.io/projected/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-kube-api-access-lv779\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.270880 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.270845 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.270919 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.270906 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.270982 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.270965 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.271033 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.270998 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.271084 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.271023 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:42.271084 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.271033 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.271187 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.271088 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.271187 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.271139 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.271327 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.271303 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.271327 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.271319 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.271582 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.271563 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.271787 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.271758 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.273122 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.273097 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.273382 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.273366 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.280343 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.280324 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv779\" (UniqueName: \"kubernetes.io/projected/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-kube-api-access-lv779\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.280593 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.280572 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f52e88-52c0-4a77-b1ff-bbb40045d05d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9s7wb\" (UID: \"d7f52e88-52c0-4a77-b1ff-bbb40045d05d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.419688 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.419657 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:42.578532 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.578508 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb"] Jul 27 21:52:42.580315 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:52:42.580287 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f52e88_52c0_4a77_b1ff_bbb40045d05d.slice/crio-37e406a7946b53d77acc713c79d0f49cf714059859606b5b6e11f627e4cf0db5 WatchSource:0}: Error finding container 37e406a7946b53d77acc713c79d0f49cf714059859606b5b6e11f627e4cf0db5: Status 404 returned error can't find the container with id 37e406a7946b53d77acc713c79d0f49cf714059859606b5b6e11f627e4cf0db5 Jul 27 21:52:42.582343 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.582311 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 21:52:42.582442 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.582383 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 21:52:42.582442 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.582427 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 21:52:42.751393 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.751354 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" event={"ID":"d7f52e88-52c0-4a77-b1ff-bbb40045d05d","Type":"ContainerStarted","Data":"aa205f9efc18ff500e0938681bcdf7674b7fcb3f5c3aaa8ca3e7c1fd0639d80f"} Jul 27 21:52:42.751549 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.751427 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" event={"ID":"d7f52e88-52c0-4a77-b1ff-bbb40045d05d","Type":"ContainerStarted","Data":"37e406a7946b53d77acc713c79d0f49cf714059859606b5b6e11f627e4cf0db5"} Jul 27 21:52:42.771125 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:42.771079 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" podStartSLOduration=0.771064337 podStartE2EDuration="771.064337ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:52:42.769435588 +0000 UTC m=+440.386872908" watchObservedRunningTime="2026-07-27 21:52:42.771064337 +0000 UTC m=+440.388501722" Jul 27 21:52:43.419935 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:43.419896 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:43.424646 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:43.424620 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:43.756730 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:43.756640 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:43.757757 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:43.757739 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:52:56.040635 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:56.040556 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:56.141540 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:56.141509 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:56.200989 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:56.200956 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:56.854559 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:56.854524 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-5g6mg"] Jul 27 21:52:56.881629 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:56.881595 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-5g6mg"] Jul 27 21:52:56.881629 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:56.881603 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5g6mg" Jul 27 21:52:56.883806 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:56.883788 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xsdrc\"" Jul 27 21:52:56.891778 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:56.891757 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:56.992914 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:56.992877 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:57.004013 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.003984 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnp56\" (UniqueName: \"kubernetes.io/projected/c6591cc1-adb2-4cb3-8ab0-de1817964d60-kube-api-access-qnp56\") pod \"authorino-58cd955675-5g6mg\" (UID: \"c6591cc1-adb2-4cb3-8ab0-de1817964d60\") " pod="kuadrant-system/authorino-58cd955675-5g6mg" Jul 27 21:52:57.058301 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.058251 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-vtf7b"] Jul 27 21:52:57.061492 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.061468 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vtf7b" Jul 27 21:52:57.069207 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.069183 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vtf7b"] Jul 27 21:52:57.105211 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.105119 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnp56\" (UniqueName: \"kubernetes.io/projected/c6591cc1-adb2-4cb3-8ab0-de1817964d60-kube-api-access-qnp56\") pod \"authorino-58cd955675-5g6mg\" (UID: \"c6591cc1-adb2-4cb3-8ab0-de1817964d60\") " pod="kuadrant-system/authorino-58cd955675-5g6mg" Jul 27 21:52:57.113947 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.113916 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnp56\" (UniqueName: \"kubernetes.io/projected/c6591cc1-adb2-4cb3-8ab0-de1817964d60-kube-api-access-qnp56\") pod \"authorino-58cd955675-5g6mg\" (UID: \"c6591cc1-adb2-4cb3-8ab0-de1817964d60\") " pod="kuadrant-system/authorino-58cd955675-5g6mg" Jul 27 21:52:57.190719 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.190674 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5g6mg" Jul 27 21:52:57.206061 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.206032 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrz68\" (UniqueName: \"kubernetes.io/projected/ef269acb-80a6-4d83-aafe-6ac3dc6a015e-kube-api-access-qrz68\") pod \"authorino-75d556bd77-vtf7b\" (UID: \"ef269acb-80a6-4d83-aafe-6ac3dc6a015e\") " pod="kuadrant-system/authorino-75d556bd77-vtf7b" Jul 27 21:52:57.306868 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.306836 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrz68\" (UniqueName: \"kubernetes.io/projected/ef269acb-80a6-4d83-aafe-6ac3dc6a015e-kube-api-access-qrz68\") pod \"authorino-75d556bd77-vtf7b\" (UID: \"ef269acb-80a6-4d83-aafe-6ac3dc6a015e\") " pod="kuadrant-system/authorino-75d556bd77-vtf7b" Jul 27 21:52:57.315433 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.315405 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrz68\" (UniqueName: \"kubernetes.io/projected/ef269acb-80a6-4d83-aafe-6ac3dc6a015e-kube-api-access-qrz68\") pod \"authorino-75d556bd77-vtf7b\" (UID: \"ef269acb-80a6-4d83-aafe-6ac3dc6a015e\") " pod="kuadrant-system/authorino-75d556bd77-vtf7b" Jul 27 21:52:57.324164 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.324134 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-5g6mg"] Jul 27 21:52:57.327443 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:52:57.327407 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6591cc1_adb2_4cb3_8ab0_de1817964d60.slice/crio-00627020da844ad004efc9555d7d65d9d7057ee0e34a42778d5022479eb8d62c WatchSource:0}: Error finding container 00627020da844ad004efc9555d7d65d9d7057ee0e34a42778d5022479eb8d62c: Status 404 returned error can't find the container with id 00627020da844ad004efc9555d7d65d9d7057ee0e34a42778d5022479eb8d62c Jul 27 21:52:57.372370 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.372289 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vtf7b" Jul 27 21:52:57.505538 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.505511 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vtf7b"] Jul 27 21:52:57.507834 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:52:57.507808 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef269acb_80a6_4d83_aafe_6ac3dc6a015e.slice/crio-9e36a6e8cc5c05bba8cb4ccc7227720f2f74a39bfb8078bc32929d1b6b344916 WatchSource:0}: Error finding container 9e36a6e8cc5c05bba8cb4ccc7227720f2f74a39bfb8078bc32929d1b6b344916: Status 404 returned error can't find the container with id 9e36a6e8cc5c05bba8cb4ccc7227720f2f74a39bfb8078bc32929d1b6b344916 Jul 27 21:52:57.831873 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.831833 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vtf7b" event={"ID":"ef269acb-80a6-4d83-aafe-6ac3dc6a015e","Type":"ContainerStarted","Data":"9e36a6e8cc5c05bba8cb4ccc7227720f2f74a39bfb8078bc32929d1b6b344916"} Jul 27 21:52:57.832990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:52:57.832961 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5g6mg" event={"ID":"c6591cc1-adb2-4cb3-8ab0-de1817964d60","Type":"ContainerStarted","Data":"00627020da844ad004efc9555d7d65d9d7057ee0e34a42778d5022479eb8d62c"} Jul 27 21:53:01.857825 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:01.857782 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5g6mg" event={"ID":"c6591cc1-adb2-4cb3-8ab0-de1817964d60","Type":"ContainerStarted","Data":"200aa168a7b8d58f5c5883ba756f604066220b94744585adf332de68413f2fdc"} Jul 27 21:53:01.859211 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:01.859188 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vtf7b" event={"ID":"ef269acb-80a6-4d83-aafe-6ac3dc6a015e","Type":"ContainerStarted","Data":"71c8d76018edd9ed999b3e76b96f2b78ddf875acd4177edac663356813cafb4d"} Jul 27 21:53:01.873223 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:01.873169 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-5g6mg" podStartSLOduration=2.292151129 podStartE2EDuration="5.873154947s" totalImagesPullingTime="3.581003818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:53:01.871795156 +0000 UTC m=+459.489232476" watchObservedRunningTime="2026-07-27 21:53:01.873154947 +0000 UTC m=+459.490592265" Jul 27 21:53:01.886986 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:01.886933 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-vtf7b" podStartSLOduration=1.474122617 podStartE2EDuration="4.886916442s" totalImagesPullingTime="3.412793825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:53:01.885591515 +0000 UTC m=+459.503028835" watchObservedRunningTime="2026-07-27 21:53:01.886916442 +0000 UTC m=+459.504353765" Jul 27 21:53:01.917287 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:01.917236 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-5g6mg"] Jul 27 21:53:03.867921 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:03.867882 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-5g6mg" podUID="c6591cc1-adb2-4cb3-8ab0-de1817964d60" containerName="authorino" containerID="cri-o://200aa168a7b8d58f5c5883ba756f604066220b94744585adf332de68413f2fdc" gracePeriod=30 Jul 27 21:53:04.115465 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:04.115439 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5g6mg" Jul 27 21:53:04.167181 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:04.167138 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnp56\" (UniqueName: \"kubernetes.io/projected/c6591cc1-adb2-4cb3-8ab0-de1817964d60-kube-api-access-qnp56\") pod \"c6591cc1-adb2-4cb3-8ab0-de1817964d60\" (UID: \"c6591cc1-adb2-4cb3-8ab0-de1817964d60\") " Jul 27 21:53:04.169197 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:04.169173 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6591cc1-adb2-4cb3-8ab0-de1817964d60-kube-api-access-qnp56" (OuterVolumeSpecName: "kube-api-access-qnp56") pod "c6591cc1-adb2-4cb3-8ab0-de1817964d60" (UID: "c6591cc1-adb2-4cb3-8ab0-de1817964d60"). InnerVolumeSpecName "kube-api-access-qnp56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:53:04.268609 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:04.268567 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnp56\" (UniqueName: \"kubernetes.io/projected/c6591cc1-adb2-4cb3-8ab0-de1817964d60-kube-api-access-qnp56\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:53:04.873658 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:04.873622 2561 generic.go:356] "Generic (PLEG): container finished" podID="c6591cc1-adb2-4cb3-8ab0-de1817964d60" containerID="200aa168a7b8d58f5c5883ba756f604066220b94744585adf332de68413f2fdc" exitCode=0 Jul 27 21:53:04.874162 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:04.873675 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5g6mg" Jul 27 21:53:04.874162 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:04.873714 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5g6mg" event={"ID":"c6591cc1-adb2-4cb3-8ab0-de1817964d60","Type":"ContainerDied","Data":"200aa168a7b8d58f5c5883ba756f604066220b94744585adf332de68413f2fdc"} Jul 27 21:53:04.874162 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:04.873760 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5g6mg" event={"ID":"c6591cc1-adb2-4cb3-8ab0-de1817964d60","Type":"ContainerDied","Data":"00627020da844ad004efc9555d7d65d9d7057ee0e34a42778d5022479eb8d62c"} Jul 27 21:53:04.874162 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:04.873782 2561 scope.go:117] "RemoveContainer" containerID="200aa168a7b8d58f5c5883ba756f604066220b94744585adf332de68413f2fdc" Jul 27 21:53:04.882692 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:04.882670 2561 scope.go:117] "RemoveContainer" containerID="200aa168a7b8d58f5c5883ba756f604066220b94744585adf332de68413f2fdc" Jul 27 21:53:04.883020 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:53:04.882997 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200aa168a7b8d58f5c5883ba756f604066220b94744585adf332de68413f2fdc\": container with ID starting with 200aa168a7b8d58f5c5883ba756f604066220b94744585adf332de68413f2fdc not found: ID does not exist" containerID="200aa168a7b8d58f5c5883ba756f604066220b94744585adf332de68413f2fdc" Jul 27 21:53:04.883097 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:04.883026 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200aa168a7b8d58f5c5883ba756f604066220b94744585adf332de68413f2fdc"} err="failed to get container status \"200aa168a7b8d58f5c5883ba756f604066220b94744585adf332de68413f2fdc\": rpc error: code = NotFound desc = could not find container \"200aa168a7b8d58f5c5883ba756f604066220b94744585adf332de68413f2fdc\": container with ID starting with 200aa168a7b8d58f5c5883ba756f604066220b94744585adf332de68413f2fdc not found: ID does not exist" Jul 27 21:53:04.892779 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:04.892742 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-5g6mg"] Jul 27 21:53:04.899451 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:04.899423 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-5g6mg"] Jul 27 21:53:06.818412 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:06.818378 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6591cc1-adb2-4cb3-8ab0-de1817964d60" path="/var/lib/kubelet/pods/c6591cc1-adb2-4cb3-8ab0-de1817964d60/volumes" Jul 27 21:53:15.225790 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:15.225753 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:15.325444 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:15.325411 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:15.735153 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:15.735125 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:23.536737 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:23.536706 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:28.348898 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.348861 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:28.582938 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.582908 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:28.583083 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.582906 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kcbdf"] Jul 27 21:53:28.583279 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.583254 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6591cc1-adb2-4cb3-8ab0-de1817964d60" containerName="authorino" Jul 27 21:53:28.583319 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.583280 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6591cc1-adb2-4cb3-8ab0-de1817964d60" containerName="authorino" Jul 27 21:53:28.583352 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.583347 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6591cc1-adb2-4cb3-8ab0-de1817964d60" containerName="authorino" Jul 27 21:53:28.587492 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.587477 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kcbdf" Jul 27 21:53:28.592257 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.592233 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kcbdf"] Jul 27 21:53:28.681429 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.681395 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zjz9\" (UniqueName: \"kubernetes.io/projected/b8b0ca16-addf-4d40-83f4-0619817edcec-kube-api-access-7zjz9\") pod \"authorino-5d4b67c446-kcbdf\" (UID: \"b8b0ca16-addf-4d40-83f4-0619817edcec\") " pod="kuadrant-system/authorino-5d4b67c446-kcbdf" Jul 27 21:53:28.782084 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.782048 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zjz9\" (UniqueName: \"kubernetes.io/projected/b8b0ca16-addf-4d40-83f4-0619817edcec-kube-api-access-7zjz9\") pod \"authorino-5d4b67c446-kcbdf\" (UID: \"b8b0ca16-addf-4d40-83f4-0619817edcec\") " pod="kuadrant-system/authorino-5d4b67c446-kcbdf" Jul 27 21:53:28.789573 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.789538 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zjz9\" (UniqueName: \"kubernetes.io/projected/b8b0ca16-addf-4d40-83f4-0619817edcec-kube-api-access-7zjz9\") pod \"authorino-5d4b67c446-kcbdf\" (UID: \"b8b0ca16-addf-4d40-83f4-0619817edcec\") " pod="kuadrant-system/authorino-5d4b67c446-kcbdf" Jul 27 21:53:28.819571 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.819543 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kcbdf"] Jul 27 21:53:28.819823 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.819808 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kcbdf" Jul 27 21:53:28.847422 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.847393 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f9955bc7d-bwzpt"] Jul 27 21:53:28.851719 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.851695 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9955bc7d-bwzpt" Jul 27 21:53:28.858287 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.858236 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f9955bc7d-bwzpt"] Jul 27 21:53:28.954747 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.954724 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kcbdf"] Jul 27 21:53:28.957027 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:53:28.957003 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8b0ca16_addf_4d40_83f4_0619817edcec.slice/crio-9fe9076f8840a36c265b7d4ec372dafbc5c03a85194185431e5f162dfecde551 WatchSource:0}: Error finding container 9fe9076f8840a36c265b7d4ec372dafbc5c03a85194185431e5f162dfecde551: Status 404 returned error can't find the container with id 9fe9076f8840a36c265b7d4ec372dafbc5c03a85194185431e5f162dfecde551 Jul 27 21:53:28.984165 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.984138 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5jvk\" (UniqueName: \"kubernetes.io/projected/cf56cd41-fd47-4d46-a9b3-c41bd82f6044-kube-api-access-g5jvk\") pod \"authorino-5f9955bc7d-bwzpt\" (UID: \"cf56cd41-fd47-4d46-a9b3-c41bd82f6044\") " pod="kuadrant-system/authorino-5f9955bc7d-bwzpt" Jul 27 21:53:28.989574 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:28.989547 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-kcbdf" event={"ID":"b8b0ca16-addf-4d40-83f4-0619817edcec","Type":"ContainerStarted","Data":"9fe9076f8840a36c265b7d4ec372dafbc5c03a85194185431e5f162dfecde551"} Jul 27 21:53:29.021277 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.021238 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f9955bc7d-bwzpt"] Jul 27 21:53:29.021537 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:53:29.021514 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-g5jvk], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5f9955bc7d-bwzpt" podUID="cf56cd41-fd47-4d46-a9b3-c41bd82f6044" Jul 27 21:53:29.048367 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.048337 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d694487c8-6rgng"] Jul 27 21:53:29.051728 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.051712 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d694487c8-6rgng" Jul 27 21:53:29.054159 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.054139 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 27 21:53:29.059360 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.059336 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d694487c8-6rgng"] Jul 27 21:53:29.084952 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.084919 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5jvk\" (UniqueName: \"kubernetes.io/projected/cf56cd41-fd47-4d46-a9b3-c41bd82f6044-kube-api-access-g5jvk\") pod \"authorino-5f9955bc7d-bwzpt\" (UID: \"cf56cd41-fd47-4d46-a9b3-c41bd82f6044\") " pod="kuadrant-system/authorino-5f9955bc7d-bwzpt" Jul 27 21:53:29.093502 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.093473 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5jvk\" (UniqueName: \"kubernetes.io/projected/cf56cd41-fd47-4d46-a9b3-c41bd82f6044-kube-api-access-g5jvk\") pod \"authorino-5f9955bc7d-bwzpt\" (UID: \"cf56cd41-fd47-4d46-a9b3-c41bd82f6044\") " pod="kuadrant-system/authorino-5f9955bc7d-bwzpt" Jul 27 21:53:29.185574 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.185542 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a990fae0-18d0-4255-8134-74cf1514b9c3-tls-cert\") pod \"authorino-7d694487c8-6rgng\" (UID: \"a990fae0-18d0-4255-8134-74cf1514b9c3\") " pod="kuadrant-system/authorino-7d694487c8-6rgng" Jul 27 21:53:29.185759 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.185598 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lqt8\" (UniqueName: \"kubernetes.io/projected/a990fae0-18d0-4255-8134-74cf1514b9c3-kube-api-access-8lqt8\") pod \"authorino-7d694487c8-6rgng\" (UID: \"a990fae0-18d0-4255-8134-74cf1514b9c3\") " pod="kuadrant-system/authorino-7d694487c8-6rgng" Jul 27 21:53:29.287047 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.286957 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a990fae0-18d0-4255-8134-74cf1514b9c3-tls-cert\") pod \"authorino-7d694487c8-6rgng\" (UID: \"a990fae0-18d0-4255-8134-74cf1514b9c3\") " pod="kuadrant-system/authorino-7d694487c8-6rgng" Jul 27 21:53:29.287047 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.287027 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lqt8\" (UniqueName: \"kubernetes.io/projected/a990fae0-18d0-4255-8134-74cf1514b9c3-kube-api-access-8lqt8\") pod \"authorino-7d694487c8-6rgng\" (UID: \"a990fae0-18d0-4255-8134-74cf1514b9c3\") " pod="kuadrant-system/authorino-7d694487c8-6rgng" Jul 27 21:53:29.289362 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.289343 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a990fae0-18d0-4255-8134-74cf1514b9c3-tls-cert\") pod \"authorino-7d694487c8-6rgng\" (UID: \"a990fae0-18d0-4255-8134-74cf1514b9c3\") " pod="kuadrant-system/authorino-7d694487c8-6rgng" Jul 27 21:53:29.294254 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.294224 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lqt8\" (UniqueName: \"kubernetes.io/projected/a990fae0-18d0-4255-8134-74cf1514b9c3-kube-api-access-8lqt8\") pod \"authorino-7d694487c8-6rgng\" (UID: \"a990fae0-18d0-4255-8134-74cf1514b9c3\") " pod="kuadrant-system/authorino-7d694487c8-6rgng" Jul 27 21:53:29.361862 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.361838 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d694487c8-6rgng" Jul 27 21:53:29.499002 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.498978 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d694487c8-6rgng"] Jul 27 21:53:29.501375 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:53:29.501341 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda990fae0_18d0_4255_8134_74cf1514b9c3.slice/crio-7f47b2d1bf75bfc23a2ac2351a13df37bc9a1b84b327ad88628f86a4e747ad0a WatchSource:0}: Error finding container 7f47b2d1bf75bfc23a2ac2351a13df37bc9a1b84b327ad88628f86a4e747ad0a: Status 404 returned error can't find the container with id 7f47b2d1bf75bfc23a2ac2351a13df37bc9a1b84b327ad88628f86a4e747ad0a Jul 27 21:53:29.996761 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.996724 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-kcbdf" event={"ID":"b8b0ca16-addf-4d40-83f4-0619817edcec","Type":"ContainerStarted","Data":"74ae47aa825528eb0e897623705ffdd643dd497e2ffd8992007bd5f9d75f6b64"} Jul 27 21:53:29.996941 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.996778 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-kcbdf" podUID="b8b0ca16-addf-4d40-83f4-0619817edcec" containerName="authorino" containerID="cri-o://74ae47aa825528eb0e897623705ffdd643dd497e2ffd8992007bd5f9d75f6b64" gracePeriod=30 Jul 27 21:53:29.998367 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.998342 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d694487c8-6rgng" event={"ID":"a990fae0-18d0-4255-8134-74cf1514b9c3","Type":"ContainerStarted","Data":"387f828d186a0ee33f5a9f8a1b87ea9cecbf14e87a6cca094e3284b3361b3f2e"} Jul 27 21:53:29.998464 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.998374 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9955bc7d-bwzpt" Jul 27 21:53:29.998518 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:29.998376 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d694487c8-6rgng" event={"ID":"a990fae0-18d0-4255-8134-74cf1514b9c3","Type":"ContainerStarted","Data":"7f47b2d1bf75bfc23a2ac2351a13df37bc9a1b84b327ad88628f86a4e747ad0a"} Jul 27 21:53:30.006176 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.006158 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9955bc7d-bwzpt" Jul 27 21:53:30.016076 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.016032 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-kcbdf" podStartSLOduration=1.633976964 podStartE2EDuration="2.016017341s" totalImagesPullingTime="382.040377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:53:30.014688184 +0000 UTC m=+487.632125505" watchObservedRunningTime="2026-07-27 21:53:30.016017341 +0000 UTC m=+487.633454659" Jul 27 21:53:30.045011 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.044962 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d694487c8-6rgng" podStartSLOduration=0.661638776 podStartE2EDuration="1.044945692s" totalImagesPullingTime="383.306916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:53:30.043148004 +0000 UTC m=+487.660585323" watchObservedRunningTime="2026-07-27 21:53:30.044945692 +0000 UTC m=+487.662383008" Jul 27 21:53:30.071903 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.071872 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vtf7b"] Jul 27 21:53:30.072078 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.072057 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-vtf7b" podUID="ef269acb-80a6-4d83-aafe-6ac3dc6a015e" containerName="authorino" containerID="cri-o://71c8d76018edd9ed999b3e76b96f2b78ddf875acd4177edac663356813cafb4d" gracePeriod=30 Jul 27 21:53:30.095234 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.095206 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5jvk\" (UniqueName: \"kubernetes.io/projected/cf56cd41-fd47-4d46-a9b3-c41bd82f6044-kube-api-access-g5jvk\") pod \"cf56cd41-fd47-4d46-a9b3-c41bd82f6044\" (UID: \"cf56cd41-fd47-4d46-a9b3-c41bd82f6044\") " Jul 27 21:53:30.097374 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.097345 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf56cd41-fd47-4d46-a9b3-c41bd82f6044-kube-api-access-g5jvk" (OuterVolumeSpecName: "kube-api-access-g5jvk") pod "cf56cd41-fd47-4d46-a9b3-c41bd82f6044" (UID: "cf56cd41-fd47-4d46-a9b3-c41bd82f6044"). InnerVolumeSpecName "kube-api-access-g5jvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:53:30.195991 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.195959 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5jvk\" (UniqueName: \"kubernetes.io/projected/cf56cd41-fd47-4d46-a9b3-c41bd82f6044-kube-api-access-g5jvk\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:53:30.300895 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.300863 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kcbdf" Jul 27 21:53:30.338084 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.338060 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vtf7b" Jul 27 21:53:30.398040 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.398012 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zjz9\" (UniqueName: \"kubernetes.io/projected/b8b0ca16-addf-4d40-83f4-0619817edcec-kube-api-access-7zjz9\") pod \"b8b0ca16-addf-4d40-83f4-0619817edcec\" (UID: \"b8b0ca16-addf-4d40-83f4-0619817edcec\") " Jul 27 21:53:30.400053 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.400030 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b0ca16-addf-4d40-83f4-0619817edcec-kube-api-access-7zjz9" (OuterVolumeSpecName: "kube-api-access-7zjz9") pod "b8b0ca16-addf-4d40-83f4-0619817edcec" (UID: "b8b0ca16-addf-4d40-83f4-0619817edcec"). InnerVolumeSpecName "kube-api-access-7zjz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:53:30.499630 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.499535 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrz68\" (UniqueName: \"kubernetes.io/projected/ef269acb-80a6-4d83-aafe-6ac3dc6a015e-kube-api-access-qrz68\") pod \"ef269acb-80a6-4d83-aafe-6ac3dc6a015e\" (UID: \"ef269acb-80a6-4d83-aafe-6ac3dc6a015e\") " Jul 27 21:53:30.499796 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.499782 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zjz9\" (UniqueName: \"kubernetes.io/projected/b8b0ca16-addf-4d40-83f4-0619817edcec-kube-api-access-7zjz9\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:53:30.501579 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.501546 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef269acb-80a6-4d83-aafe-6ac3dc6a015e-kube-api-access-qrz68" (OuterVolumeSpecName: "kube-api-access-qrz68") pod "ef269acb-80a6-4d83-aafe-6ac3dc6a015e" (UID: "ef269acb-80a6-4d83-aafe-6ac3dc6a015e"). InnerVolumeSpecName "kube-api-access-qrz68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:53:30.600760 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:30.600710 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrz68\" (UniqueName: \"kubernetes.io/projected/ef269acb-80a6-4d83-aafe-6ac3dc6a015e-kube-api-access-qrz68\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:53:31.003582 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.003550 2561 generic.go:356] "Generic (PLEG): container finished" podID="b8b0ca16-addf-4d40-83f4-0619817edcec" containerID="74ae47aa825528eb0e897623705ffdd643dd497e2ffd8992007bd5f9d75f6b64" exitCode=0 Jul 27 21:53:31.003750 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.003608 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kcbdf" Jul 27 21:53:31.003750 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.003628 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-kcbdf" event={"ID":"b8b0ca16-addf-4d40-83f4-0619817edcec","Type":"ContainerDied","Data":"74ae47aa825528eb0e897623705ffdd643dd497e2ffd8992007bd5f9d75f6b64"} Jul 27 21:53:31.003750 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.003673 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-kcbdf" event={"ID":"b8b0ca16-addf-4d40-83f4-0619817edcec","Type":"ContainerDied","Data":"9fe9076f8840a36c265b7d4ec372dafbc5c03a85194185431e5f162dfecde551"} Jul 27 21:53:31.003750 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.003692 2561 scope.go:117] "RemoveContainer" containerID="74ae47aa825528eb0e897623705ffdd643dd497e2ffd8992007bd5f9d75f6b64" Jul 27 21:53:31.005062 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.004948 2561 generic.go:356] "Generic (PLEG): container finished" podID="ef269acb-80a6-4d83-aafe-6ac3dc6a015e" containerID="71c8d76018edd9ed999b3e76b96f2b78ddf875acd4177edac663356813cafb4d" exitCode=0 Jul 27 21:53:31.005062 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.005019 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vtf7b" Jul 27 21:53:31.005062 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.005032 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9955bc7d-bwzpt" Jul 27 21:53:31.005062 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.005037 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vtf7b" event={"ID":"ef269acb-80a6-4d83-aafe-6ac3dc6a015e","Type":"ContainerDied","Data":"71c8d76018edd9ed999b3e76b96f2b78ddf875acd4177edac663356813cafb4d"} Jul 27 21:53:31.005300 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.005071 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vtf7b" event={"ID":"ef269acb-80a6-4d83-aafe-6ac3dc6a015e","Type":"ContainerDied","Data":"9e36a6e8cc5c05bba8cb4ccc7227720f2f74a39bfb8078bc32929d1b6b344916"} Jul 27 21:53:31.012508 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.012492 2561 scope.go:117] "RemoveContainer" containerID="74ae47aa825528eb0e897623705ffdd643dd497e2ffd8992007bd5f9d75f6b64" Jul 27 21:53:31.012772 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:53:31.012754 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ae47aa825528eb0e897623705ffdd643dd497e2ffd8992007bd5f9d75f6b64\": container with ID starting with 74ae47aa825528eb0e897623705ffdd643dd497e2ffd8992007bd5f9d75f6b64 not found: ID does not exist" containerID="74ae47aa825528eb0e897623705ffdd643dd497e2ffd8992007bd5f9d75f6b64" Jul 27 21:53:31.012816 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.012779 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ae47aa825528eb0e897623705ffdd643dd497e2ffd8992007bd5f9d75f6b64"} err="failed to get container status \"74ae47aa825528eb0e897623705ffdd643dd497e2ffd8992007bd5f9d75f6b64\": rpc error: code = NotFound desc = could not find container \"74ae47aa825528eb0e897623705ffdd643dd497e2ffd8992007bd5f9d75f6b64\": container with ID starting with 74ae47aa825528eb0e897623705ffdd643dd497e2ffd8992007bd5f9d75f6b64 not found: ID does not exist" Jul 27 21:53:31.012816 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.012797 2561 scope.go:117] "RemoveContainer" containerID="71c8d76018edd9ed999b3e76b96f2b78ddf875acd4177edac663356813cafb4d" Jul 27 21:53:31.020035 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.020018 2561 scope.go:117] "RemoveContainer" containerID="71c8d76018edd9ed999b3e76b96f2b78ddf875acd4177edac663356813cafb4d" Jul 27 21:53:31.020306 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:53:31.020285 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c8d76018edd9ed999b3e76b96f2b78ddf875acd4177edac663356813cafb4d\": container with ID starting with 71c8d76018edd9ed999b3e76b96f2b78ddf875acd4177edac663356813cafb4d not found: ID does not exist" containerID="71c8d76018edd9ed999b3e76b96f2b78ddf875acd4177edac663356813cafb4d" Jul 27 21:53:31.020390 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.020309 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c8d76018edd9ed999b3e76b96f2b78ddf875acd4177edac663356813cafb4d"} err="failed to get container status \"71c8d76018edd9ed999b3e76b96f2b78ddf875acd4177edac663356813cafb4d\": rpc error: code = NotFound desc = could not find container \"71c8d76018edd9ed999b3e76b96f2b78ddf875acd4177edac663356813cafb4d\": container with ID starting with 71c8d76018edd9ed999b3e76b96f2b78ddf875acd4177edac663356813cafb4d not found: ID does not exist" Jul 27 21:53:31.026024 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.026002 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kcbdf"] Jul 27 21:53:31.029700 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.029667 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kcbdf"] Jul 27 21:53:31.051059 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.050880 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f9955bc7d-bwzpt"] Jul 27 21:53:31.054466 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.054443 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f9955bc7d-bwzpt"] Jul 27 21:53:31.062557 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.062510 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vtf7b"] Jul 27 21:53:31.064286 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:31.064251 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vtf7b"] Jul 27 21:53:32.716351 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:32.716312 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:32.818101 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:32.818070 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b0ca16-addf-4d40-83f4-0619817edcec" path="/var/lib/kubelet/pods/b8b0ca16-addf-4d40-83f4-0619817edcec/volumes" Jul 27 21:53:32.818424 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:32.818412 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf56cd41-fd47-4d46-a9b3-c41bd82f6044" path="/var/lib/kubelet/pods/cf56cd41-fd47-4d46-a9b3-c41bd82f6044/volumes" Jul 27 21:53:32.818606 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:32.818595 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef269acb-80a6-4d83-aafe-6ac3dc6a015e" path="/var/lib/kubelet/pods/ef269acb-80a6-4d83-aafe-6ac3dc6a015e/volumes" Jul 27 21:53:34.773510 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:34.773479 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:34.862658 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:34.862627 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:34.950990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:34.950957 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:35.363773 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:35.363742 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:38.479782 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:38.479749 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:38.584695 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:53:38.584660 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:54:11.188762 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:11.188714 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:54:11.277141 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:11.277107 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:54:11.800689 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:11.800653 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:54:11.900683 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:11.900643 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:54:12.404238 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:12.404206 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:54:12.498507 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:12.498476 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:54:20.251057 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:20.250958 2561 generic.go:356] "Generic (PLEG): container finished" podID="d7f52e88-52c0-4a77-b1ff-bbb40045d05d" containerID="aa205f9efc18ff500e0938681bcdf7674b7fcb3f5c3aaa8ca3e7c1fd0639d80f" exitCode=0 Jul 27 21:54:20.251057 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:20.251034 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" event={"ID":"d7f52e88-52c0-4a77-b1ff-bbb40045d05d","Type":"ContainerDied","Data":"aa205f9efc18ff500e0938681bcdf7674b7fcb3f5c3aaa8ca3e7c1fd0639d80f"} Jul 27 21:54:20.251540 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:20.251448 2561 scope.go:117] "RemoveContainer" containerID="aa205f9efc18ff500e0938681bcdf7674b7fcb3f5c3aaa8ca3e7c1fd0639d80f" Jul 27 21:54:20.252479 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:20.252455 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 21:54:20.252557 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:20.252517 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 21:54:20.252602 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:20.252558 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 21:54:21.260217 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:21.260182 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" event={"ID":"d7f52e88-52c0-4a77-b1ff-bbb40045d05d","Type":"ContainerStarted","Data":"baf9e33c19507f15856ada508dee190b32835cde778016fbdfee7d829ef134a3"} Jul 27 21:54:21.420034 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:21.419997 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:54:21.420306 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:21.420246 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-9s7wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 27 21:54:21.420361 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:21.420313 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" podUID="d7f52e88-52c0-4a77-b1ff-bbb40045d05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 27 21:54:22.277170 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:22.277133 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:54:22.368037 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:22.368005 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:54:22.420456 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:22.420418 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-9s7wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 27 21:54:22.420622 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:22.420476 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" podUID="d7f52e88-52c0-4a77-b1ff-bbb40045d05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 27 21:54:23.420300 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:23.420249 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-9s7wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 27 21:54:23.420688 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:23.420322 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" podUID="d7f52e88-52c0-4a77-b1ff-bbb40045d05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 27 21:54:24.420401 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:24.420361 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-9s7wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 27 21:54:24.420877 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:24.420439 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" podUID="d7f52e88-52c0-4a77-b1ff-bbb40045d05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 27 21:54:25.420206 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:25.420169 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-9s7wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 27 21:54:25.420414 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:25.420225 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" podUID="d7f52e88-52c0-4a77-b1ff-bbb40045d05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 27 21:54:26.423848 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:26.423819 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:54:26.424231 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:26.424097 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:54:26.424819 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:26.424800 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9s7wb" Jul 27 21:54:27.365726 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:27.365696 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:54:31.175326 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.175294 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh"] Jul 27 21:54:31.175718 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.175629 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef269acb-80a6-4d83-aafe-6ac3dc6a015e" containerName="authorino" Jul 27 21:54:31.175718 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.175640 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef269acb-80a6-4d83-aafe-6ac3dc6a015e" containerName="authorino" Jul 27 21:54:31.175718 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.175657 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b0ca16-addf-4d40-83f4-0619817edcec" containerName="authorino" Jul 27 21:54:31.175718 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.175662 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b0ca16-addf-4d40-83f4-0619817edcec" containerName="authorino" Jul 27 21:54:31.175718 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.175711 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef269acb-80a6-4d83-aafe-6ac3dc6a015e" containerName="authorino" Jul 27 21:54:31.175718 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.175722 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b0ca16-addf-4d40-83f4-0619817edcec" containerName="authorino" Jul 27 21:54:31.179112 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.179095 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.182964 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.182941 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 27 21:54:31.183102 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.183034 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 27 21:54:31.183102 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.183048 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h4jdf\"" Jul 27 21:54:31.183209 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.183063 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 27 21:54:31.195728 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.195701 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh"] Jul 27 21:54:31.251573 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.251540 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:54:31.306506 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.306472 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4226680-c358-4312-b3fe-142a709a11bb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.306690 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.306523 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4226680-c358-4312-b3fe-142a709a11bb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.306690 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.306574 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4226680-c358-4312-b3fe-142a709a11bb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.306690 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.306616 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4226680-c358-4312-b3fe-142a709a11bb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.306690 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.306671 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4226680-c358-4312-b3fe-142a709a11bb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.306850 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.306709 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77tm6\" (UniqueName: \"kubernetes.io/projected/a4226680-c358-4312-b3fe-142a709a11bb-kube-api-access-77tm6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.407613 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.407574 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4226680-c358-4312-b3fe-142a709a11bb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.407804 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.407622 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77tm6\" (UniqueName: \"kubernetes.io/projected/a4226680-c358-4312-b3fe-142a709a11bb-kube-api-access-77tm6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.407804 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.407661 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4226680-c358-4312-b3fe-142a709a11bb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.407804 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.407685 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4226680-c358-4312-b3fe-142a709a11bb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.407804 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.407702 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4226680-c358-4312-b3fe-142a709a11bb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.407804 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.407734 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4226680-c358-4312-b3fe-142a709a11bb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.408129 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.408108 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4226680-c358-4312-b3fe-142a709a11bb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.408190 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.408152 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4226680-c358-4312-b3fe-142a709a11bb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.408240 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.408187 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4226680-c358-4312-b3fe-142a709a11bb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.409990 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.409969 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4226680-c358-4312-b3fe-142a709a11bb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.410157 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.410142 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4226680-c358-4312-b3fe-142a709a11bb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.415280 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.415248 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77tm6\" (UniqueName: \"kubernetes.io/projected/a4226680-c358-4312-b3fe-142a709a11bb-kube-api-access-77tm6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lhrgh\" (UID: \"a4226680-c358-4312-b3fe-142a709a11bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.489251 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.489161 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:31.623964 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:31.623937 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh"] Jul 27 21:54:31.625503 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:54:31.625475 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4226680_c358_4312_b3fe_142a709a11bb.slice/crio-d6979337503baf03f9c44c84704b65993371799c9731e2e9288c3d96c2f0abe6 WatchSource:0}: Error finding container d6979337503baf03f9c44c84704b65993371799c9731e2e9288c3d96c2f0abe6: Status 404 returned error can't find the container with id d6979337503baf03f9c44c84704b65993371799c9731e2e9288c3d96c2f0abe6 Jul 27 21:54:32.317455 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:32.317413 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" event={"ID":"a4226680-c358-4312-b3fe-142a709a11bb","Type":"ContainerStarted","Data":"d6979337503baf03f9c44c84704b65993371799c9731e2e9288c3d96c2f0abe6"} Jul 27 21:54:33.324328 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:33.324293 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" event={"ID":"a4226680-c358-4312-b3fe-142a709a11bb","Type":"ContainerStarted","Data":"e583ba78c03472043b3751412ccaa2231b9653cd69d4c2385babbce9f09b8a60"} Jul 27 21:54:33.344172 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:33.344121 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" podStartSLOduration=1.241769472 podStartE2EDuration="2.344107752s" totalImagesPullingTime="1.10233828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:33.341909907 +0000 UTC m=+550.959347229" watchObservedRunningTime="2026-07-27 21:54:33.344107752 +0000 UTC m=+550.961545070" Jul 27 21:54:33.880843 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:33.880810 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:54:38.536895 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:38.536845 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:54:41.489794 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:41.489731 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:41.490198 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:41.489819 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:41.503419 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:41.503392 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:42.379869 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:42.379842 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lhrgh" Jul 27 21:54:46.322633 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:54:46.322601 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:04.032711 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.032675 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd"] Jul 27 21:55:04.037694 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.037668 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.039832 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.039816 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 27 21:55:04.046228 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.046203 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd"] Jul 27 21:55:04.062418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.062393 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7778340f-8115-4158-871d-19cfc605562c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.062583 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.062427 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctvn\" (UniqueName: \"kubernetes.io/projected/7778340f-8115-4158-871d-19cfc605562c-kube-api-access-sctvn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.062583 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.062488 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7778340f-8115-4158-871d-19cfc605562c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.062583 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.062553 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7778340f-8115-4158-871d-19cfc605562c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.062737 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.062609 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7778340f-8115-4158-871d-19cfc605562c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.062737 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.062635 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7778340f-8115-4158-871d-19cfc605562c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.067926 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.067905 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:04.163885 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.163851 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7778340f-8115-4158-871d-19cfc605562c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.163885 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.163889 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7778340f-8115-4158-871d-19cfc605562c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.164124 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.163918 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7778340f-8115-4158-871d-19cfc605562c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.164124 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.164031 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7778340f-8115-4158-871d-19cfc605562c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.164241 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.164158 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7778340f-8115-4158-871d-19cfc605562c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.164241 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.164199 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sctvn\" (UniqueName: \"kubernetes.io/projected/7778340f-8115-4158-871d-19cfc605562c-kube-api-access-sctvn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.164372 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.164356 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7778340f-8115-4158-871d-19cfc605562c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.164427 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.164394 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7778340f-8115-4158-871d-19cfc605562c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.164598 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.164561 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7778340f-8115-4158-871d-19cfc605562c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.166256 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.166236 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7778340f-8115-4158-871d-19cfc605562c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.166546 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.166526 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7778340f-8115-4158-871d-19cfc605562c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.171612 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.171591 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctvn\" (UniqueName: \"kubernetes.io/projected/7778340f-8115-4158-871d-19cfc605562c-kube-api-access-sctvn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d2hmd\" (UID: \"7778340f-8115-4158-871d-19cfc605562c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.349197 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.349106 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:04.478252 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:04.478227 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd"] Jul 27 21:55:04.479849 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:55:04.479824 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7778340f_8115_4158_871d_19cfc605562c.slice/crio-2fde36826ede9685be0bea6c40b914d8ecc356db202b1f324162cbe3086589ae WatchSource:0}: Error finding container 2fde36826ede9685be0bea6c40b914d8ecc356db202b1f324162cbe3086589ae: Status 404 returned error can't find the container with id 2fde36826ede9685be0bea6c40b914d8ecc356db202b1f324162cbe3086589ae Jul 27 21:55:05.484116 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:05.484079 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" event={"ID":"7778340f-8115-4158-871d-19cfc605562c","Type":"ContainerStarted","Data":"31820e0e9f788e6cbcfed3015dacc944c292cfd12edb2837bf20f9f02ec89b6f"} Jul 27 21:55:05.484617 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:05.484128 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" event={"ID":"7778340f-8115-4158-871d-19cfc605562c","Type":"ContainerStarted","Data":"2fde36826ede9685be0bea6c40b914d8ecc356db202b1f324162cbe3086589ae"} Jul 27 21:55:05.503136 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:05.503083 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" podStartSLOduration=1.317351327 podStartE2EDuration="1.503067709s" totalImagesPullingTime="185.716382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:05.501732549 +0000 UTC m=+583.119169880" watchObservedRunningTime="2026-07-27 21:55:05.503067709 +0000 UTC m=+583.120505027" Jul 27 21:55:14.349834 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:14.349802 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:14.350317 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:14.349860 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:14.362520 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:14.362489 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:14.543517 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:14.543486 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d2hmd" Jul 27 21:55:19.672229 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:55:19.672197 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:08.492982 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:08.492949 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:08.591934 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:08.591903 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:20.740646 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:20.740609 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-796c9964dc-dxhvw"] Jul 27 21:56:20.743873 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:20.743854 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-796c9964dc-dxhvw" Jul 27 21:56:20.753893 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:20.753867 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-796c9964dc-dxhvw"] Jul 27 21:56:20.837038 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:20.836990 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d3aec408-771b-409f-b46d-02197bf53a7c-tls-cert\") pod \"authorino-796c9964dc-dxhvw\" (UID: \"d3aec408-771b-409f-b46d-02197bf53a7c\") " pod="kuadrant-system/authorino-796c9964dc-dxhvw" Jul 27 21:56:20.837221 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:20.837058 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq8dt\" (UniqueName: \"kubernetes.io/projected/d3aec408-771b-409f-b46d-02197bf53a7c-kube-api-access-lq8dt\") pod \"authorino-796c9964dc-dxhvw\" (UID: \"d3aec408-771b-409f-b46d-02197bf53a7c\") " pod="kuadrant-system/authorino-796c9964dc-dxhvw" Jul 27 21:56:20.938205 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:20.938155 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d3aec408-771b-409f-b46d-02197bf53a7c-tls-cert\") pod \"authorino-796c9964dc-dxhvw\" (UID: \"d3aec408-771b-409f-b46d-02197bf53a7c\") " pod="kuadrant-system/authorino-796c9964dc-dxhvw" Jul 27 21:56:20.938418 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:20.938223 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq8dt\" (UniqueName: \"kubernetes.io/projected/d3aec408-771b-409f-b46d-02197bf53a7c-kube-api-access-lq8dt\") pod \"authorino-796c9964dc-dxhvw\" (UID: \"d3aec408-771b-409f-b46d-02197bf53a7c\") " pod="kuadrant-system/authorino-796c9964dc-dxhvw" Jul 27 21:56:20.940616 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:20.940596 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d3aec408-771b-409f-b46d-02197bf53a7c-tls-cert\") pod \"authorino-796c9964dc-dxhvw\" (UID: \"d3aec408-771b-409f-b46d-02197bf53a7c\") " pod="kuadrant-system/authorino-796c9964dc-dxhvw" Jul 27 21:56:20.947684 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:20.947655 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq8dt\" (UniqueName: \"kubernetes.io/projected/d3aec408-771b-409f-b46d-02197bf53a7c-kube-api-access-lq8dt\") pod \"authorino-796c9964dc-dxhvw\" (UID: \"d3aec408-771b-409f-b46d-02197bf53a7c\") " pod="kuadrant-system/authorino-796c9964dc-dxhvw" Jul 27 21:56:21.053031 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:21.052941 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-796c9964dc-dxhvw" Jul 27 21:56:21.201384 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:21.201355 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-796c9964dc-dxhvw"] Jul 27 21:56:21.203112 ip-10-0-129-80 kubenswrapper[2561]: W0727 21:56:21.203084 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3aec408_771b_409f_b46d_02197bf53a7c.slice/crio-15d324cf028e84649dfc9f3c39fe78fd1dd674d83a64460f6282e69bc05b6021 WatchSource:0}: Error finding container 15d324cf028e84649dfc9f3c39fe78fd1dd674d83a64460f6282e69bc05b6021: Status 404 returned error can't find the container with id 15d324cf028e84649dfc9f3c39fe78fd1dd674d83a64460f6282e69bc05b6021 Jul 27 21:56:21.204491 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:21.204473 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 21:56:21.225321 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:21.225287 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:21.327531 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:21.327425 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:21.870324 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:21.870043 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-796c9964dc-dxhvw" event={"ID":"d3aec408-771b-409f-b46d-02197bf53a7c","Type":"ContainerStarted","Data":"21bc9b709db40a1266119be6b8f9557dc7e1824f8692765fdb82c09dd61ff493"} Jul 27 21:56:21.870324 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:21.870093 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-796c9964dc-dxhvw" event={"ID":"d3aec408-771b-409f-b46d-02197bf53a7c","Type":"ContainerStarted","Data":"15d324cf028e84649dfc9f3c39fe78fd1dd674d83a64460f6282e69bc05b6021"} Jul 27 21:56:21.944058 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:21.943942 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-796c9964dc-dxhvw" podStartSLOduration=1.490596522 podStartE2EDuration="1.943915079s" totalImagesPullingTime="453.318557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:56:21.911819167 +0000 UTC m=+659.529256486" watchObservedRunningTime="2026-07-27 21:56:21.943915079 +0000 UTC m=+659.561352399" Jul 27 21:56:21.946680 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:21.946651 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d694487c8-6rgng"] Jul 27 21:56:21.946890 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:21.946860 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7d694487c8-6rgng" podUID="a990fae0-18d0-4255-8134-74cf1514b9c3" containerName="authorino" containerID="cri-o://387f828d186a0ee33f5a9f8a1b87ea9cecbf14e87a6cca094e3284b3361b3f2e" gracePeriod=30 Jul 27 21:56:22.211323 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.211296 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d694487c8-6rgng" Jul 27 21:56:22.352610 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.352571 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a990fae0-18d0-4255-8134-74cf1514b9c3-tls-cert\") pod \"a990fae0-18d0-4255-8134-74cf1514b9c3\" (UID: \"a990fae0-18d0-4255-8134-74cf1514b9c3\") " Jul 27 21:56:22.352789 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.352676 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lqt8\" (UniqueName: \"kubernetes.io/projected/a990fae0-18d0-4255-8134-74cf1514b9c3-kube-api-access-8lqt8\") pod \"a990fae0-18d0-4255-8134-74cf1514b9c3\" (UID: \"a990fae0-18d0-4255-8134-74cf1514b9c3\") " Jul 27 21:56:22.354721 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.354694 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a990fae0-18d0-4255-8134-74cf1514b9c3-kube-api-access-8lqt8" (OuterVolumeSpecName: "kube-api-access-8lqt8") pod "a990fae0-18d0-4255-8134-74cf1514b9c3" (UID: "a990fae0-18d0-4255-8134-74cf1514b9c3"). InnerVolumeSpecName "kube-api-access-8lqt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:56:22.362797 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.362767 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a990fae0-18d0-4255-8134-74cf1514b9c3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a990fae0-18d0-4255-8134-74cf1514b9c3" (UID: "a990fae0-18d0-4255-8134-74cf1514b9c3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:56:22.437185 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.437149 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:22.454238 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.454158 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lqt8\" (UniqueName: \"kubernetes.io/projected/a990fae0-18d0-4255-8134-74cf1514b9c3-kube-api-access-8lqt8\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:56:22.454238 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.454184 2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a990fae0-18d0-4255-8134-74cf1514b9c3-tls-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 21:56:22.532942 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.532907 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:22.876541 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.876501 2561 generic.go:356] "Generic (PLEG): container finished" podID="a990fae0-18d0-4255-8134-74cf1514b9c3" containerID="387f828d186a0ee33f5a9f8a1b87ea9cecbf14e87a6cca094e3284b3361b3f2e" exitCode=0 Jul 27 21:56:22.876969 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.876547 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d694487c8-6rgng" Jul 27 21:56:22.876969 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.876590 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d694487c8-6rgng" event={"ID":"a990fae0-18d0-4255-8134-74cf1514b9c3","Type":"ContainerDied","Data":"387f828d186a0ee33f5a9f8a1b87ea9cecbf14e87a6cca094e3284b3361b3f2e"} Jul 27 21:56:22.876969 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.876640 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d694487c8-6rgng" event={"ID":"a990fae0-18d0-4255-8134-74cf1514b9c3","Type":"ContainerDied","Data":"7f47b2d1bf75bfc23a2ac2351a13df37bc9a1b84b327ad88628f86a4e747ad0a"} Jul 27 21:56:22.876969 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.876664 2561 scope.go:117] "RemoveContainer" containerID="387f828d186a0ee33f5a9f8a1b87ea9cecbf14e87a6cca094e3284b3361b3f2e" Jul 27 21:56:22.889774 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.889749 2561 scope.go:117] "RemoveContainer" containerID="387f828d186a0ee33f5a9f8a1b87ea9cecbf14e87a6cca094e3284b3361b3f2e" Jul 27 21:56:22.890231 ip-10-0-129-80 kubenswrapper[2561]: E0727 21:56:22.890204 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"387f828d186a0ee33f5a9f8a1b87ea9cecbf14e87a6cca094e3284b3361b3f2e\": container with ID starting with 387f828d186a0ee33f5a9f8a1b87ea9cecbf14e87a6cca094e3284b3361b3f2e not found: ID does not exist" containerID="387f828d186a0ee33f5a9f8a1b87ea9cecbf14e87a6cca094e3284b3361b3f2e" Jul 27 21:56:22.890350 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.890237 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387f828d186a0ee33f5a9f8a1b87ea9cecbf14e87a6cca094e3284b3361b3f2e"} err="failed to get container status \"387f828d186a0ee33f5a9f8a1b87ea9cecbf14e87a6cca094e3284b3361b3f2e\": rpc error: code = NotFound desc = could not find container \"387f828d186a0ee33f5a9f8a1b87ea9cecbf14e87a6cca094e3284b3361b3f2e\": container with ID starting with 387f828d186a0ee33f5a9f8a1b87ea9cecbf14e87a6cca094e3284b3361b3f2e not found: ID does not exist" Jul 27 21:56:22.903214 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.903183 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d694487c8-6rgng"] Jul 27 21:56:22.910995 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:22.910958 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d694487c8-6rgng"] Jul 27 21:56:24.818096 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:24.818064 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a990fae0-18d0-4255-8134-74cf1514b9c3" path="/var/lib/kubelet/pods/a990fae0-18d0-4255-8134-74cf1514b9c3/volumes" Jul 27 21:56:38.571515 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:38.571477 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:38.673238 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:56:38.673210 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:18.305485 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:58:18.305420 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:19.006436 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:58:19.006396 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:19.057494 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:58:19.057461 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:28.972549 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:58:28.972498 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:30.248495 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:58:30.248462 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:42.056069 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:58:42.056005 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:42.118405 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:58:42.118370 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:05.889082 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:59:05.889042 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:05.986043 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:59:05.986013 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:06.099986 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:59:06.099954 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:12.092051 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:59:12.092019 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:12.180771 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:59:12.180732 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:12.385828 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:59:12.385739 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:18.197795 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:59:18.197763 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:18.297111 ip-10-0-129-80 kubenswrapper[2561]: I0727 21:59:18.297075 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:15.372203 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:01:15.372167 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:58.484597 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:01:58.484509 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:57.423344 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:04:57.423309 2561 generic.go:356] "Generic (PLEG): container finished" podID="e73b8505-9c44-40f5-a50a-297a426ebb3e" containerID="0e916f00dff6ba7321aae61b61998b10b0c51b588ebf3eb4c7c81bf9a68ea846" exitCode=1 Jul 27 22:04:57.423767 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:04:57.423385 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" event={"ID":"e73b8505-9c44-40f5-a50a-297a426ebb3e","Type":"ContainerDied","Data":"0e916f00dff6ba7321aae61b61998b10b0c51b588ebf3eb4c7c81bf9a68ea846"} Jul 27 22:04:57.423767 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:04:57.423725 2561 scope.go:117] "RemoveContainer" containerID="0e916f00dff6ba7321aae61b61998b10b0c51b588ebf3eb4c7c81bf9a68ea846" Jul 27 22:04:58.431606 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:04:58.431570 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" event={"ID":"e73b8505-9c44-40f5-a50a-297a426ebb3e","Type":"ContainerStarted","Data":"3c52fa6d16e76edb535fc1fe306512295d63016d1a51284b0040fecc752693ce"} Jul 27 22:04:58.432101 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:04:58.431770 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 22:05:09.439570 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:05:09.439529 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-7tdxs" Jul 27 22:06:29.972840 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:06:29.972751 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:06:31.730448 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:06:31.730410 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:03.786432 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:03.786378 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:07.239224 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.239188 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz"] Jul 27 22:07:07.239639 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.239518 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a990fae0-18d0-4255-8134-74cf1514b9c3" containerName="authorino" Jul 27 22:07:07.239639 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.239530 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a990fae0-18d0-4255-8134-74cf1514b9c3" containerName="authorino" Jul 27 22:07:07.239639 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.239590 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a990fae0-18d0-4255-8134-74cf1514b9c3" containerName="authorino" Jul 27 22:07:07.242581 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.242561 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.244809 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.244786 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e7af719f-kserve-self-signed-certs\"" Jul 27 22:07:07.252986 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.252966 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz"] Jul 27 22:07:07.295713 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.295675 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-model-cache\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.295893 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.295719 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-dshm\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.295893 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.295772 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-home\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.295893 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.295848 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-tmp-dir\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.295893 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.295880 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trc6s\" (UniqueName: \"kubernetes.io/projected/024f42cf-75c3-4c8d-8727-c3e07ff391a6-kube-api-access-trc6s\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.296050 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.295909 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024f42cf-75c3-4c8d-8727-c3e07ff391a6-tls-certs\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.340028 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.339971 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:07.396508 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.396475 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-tmp-dir\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.396508 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.396511 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trc6s\" (UniqueName: \"kubernetes.io/projected/024f42cf-75c3-4c8d-8727-c3e07ff391a6-kube-api-access-trc6s\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.396730 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.396541 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024f42cf-75c3-4c8d-8727-c3e07ff391a6-tls-certs\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.396730 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.396591 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-model-cache\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.396730 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.396608 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-dshm\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.396730 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.396635 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-home\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.396984 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.396959 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-tmp-dir\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.397045 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.396970 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-model-cache\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.397045 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.397025 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-home\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.398893 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.398864 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-dshm\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.399133 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.399115 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024f42cf-75c3-4c8d-8727-c3e07ff391a6-tls-certs\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.404295 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.404253 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trc6s\" (UniqueName: \"kubernetes.io/projected/024f42cf-75c3-4c8d-8727-c3e07ff391a6-kube-api-access-trc6s\") pod \"e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.552732 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.552639 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:07.690033 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:07.690004 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz"] Jul 27 22:07:07.692196 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:07:07.692170 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod024f42cf_75c3_4c8d_8727_c3e07ff391a6.slice/crio-40e744b1aea80b72d10597fbc032794c06584f4504a6a77ded7ddd9a0ac8a51c WatchSource:0}: Error finding container 40e744b1aea80b72d10597fbc032794c06584f4504a6a77ded7ddd9a0ac8a51c: Status 404 returned error can't find the container with id 40e744b1aea80b72d10597fbc032794c06584f4504a6a77ded7ddd9a0ac8a51c Jul 27 22:07:08.075317 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:08.075253 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" event={"ID":"024f42cf-75c3-4c8d-8727-c3e07ff391a6","Type":"ContainerStarted","Data":"798f467b25d92a9577cf8416fc769ebe502f06b782ea146b71c12eb23ac5273c"} Jul 27 22:07:08.075504 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:08.075324 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" event={"ID":"024f42cf-75c3-4c8d-8727-c3e07ff391a6","Type":"ContainerStarted","Data":"40e744b1aea80b72d10597fbc032794c06584f4504a6a77ded7ddd9a0ac8a51c"} Jul 27 22:07:08.094078 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:08.094026 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" podStartSLOduration=1.094008736 podStartE2EDuration="1.094008736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:07:08.091752894 +0000 UTC m=+1305.709190214" watchObservedRunningTime="2026-07-27 22:07:08.094008736 +0000 UTC m=+1305.711446056" Jul 27 22:07:17.553249 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:17.553211 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:17.553717 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:17.553279 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:17.565901 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:17.565866 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:18.140947 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:18.140912 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:31.288914 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:31.288877 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:32.988427 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:32.988388 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz"] Jul 27 22:07:33.013547 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:33.013508 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:48.007503 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.007407 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" podUID="024f42cf-75c3-4c8d-8727-c3e07ff391a6" containerName="main" containerID="cri-o://798f467b25d92a9577cf8416fc769ebe502f06b782ea146b71c12eb23ac5273c" gracePeriod=45 Jul 27 22:07:48.131060 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.131020 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" podUID="024f42cf-75c3-4c8d-8727-c3e07ff391a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": dial tcp 10.133.0.44:8000: connect: connection refused" Jul 27 22:07:48.251290 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.251242 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:48.274098 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.274018 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-model-cache\") pod \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " Jul 27 22:07:48.274098 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.274075 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024f42cf-75c3-4c8d-8727-c3e07ff391a6-tls-certs\") pod \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " Jul 27 22:07:48.274335 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.274110 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-home\") pod \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " Jul 27 22:07:48.274335 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.274173 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trc6s\" (UniqueName: \"kubernetes.io/projected/024f42cf-75c3-4c8d-8727-c3e07ff391a6-kube-api-access-trc6s\") pod \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " Jul 27 22:07:48.274335 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.274204 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-dshm\") pod \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " Jul 27 22:07:48.274335 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.274247 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-tmp-dir\") pod \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\" (UID: \"024f42cf-75c3-4c8d-8727-c3e07ff391a6\") " Jul 27 22:07:48.274335 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.274285 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "024f42cf-75c3-4c8d-8727-c3e07ff391a6" (UID: "024f42cf-75c3-4c8d-8727-c3e07ff391a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:07:48.274579 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.274458 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-home" (OuterVolumeSpecName: "home") pod "024f42cf-75c3-4c8d-8727-c3e07ff391a6" (UID: "024f42cf-75c3-4c8d-8727-c3e07ff391a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:07:48.274579 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.274564 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-model-cache\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:07:48.274682 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.274587 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-home\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:07:48.274682 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.274658 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "024f42cf-75c3-4c8d-8727-c3e07ff391a6" (UID: "024f42cf-75c3-4c8d-8727-c3e07ff391a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:07:48.276610 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.276578 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024f42cf-75c3-4c8d-8727-c3e07ff391a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "024f42cf-75c3-4c8d-8727-c3e07ff391a6" (UID: "024f42cf-75c3-4c8d-8727-c3e07ff391a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:07:48.276765 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.276726 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-dshm" (OuterVolumeSpecName: "dshm") pod "024f42cf-75c3-4c8d-8727-c3e07ff391a6" (UID: "024f42cf-75c3-4c8d-8727-c3e07ff391a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:07:48.276832 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.276779 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024f42cf-75c3-4c8d-8727-c3e07ff391a6-kube-api-access-trc6s" (OuterVolumeSpecName: "kube-api-access-trc6s") pod "024f42cf-75c3-4c8d-8727-c3e07ff391a6" (UID: "024f42cf-75c3-4c8d-8727-c3e07ff391a6"). InnerVolumeSpecName "kube-api-access-trc6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:07:48.286335 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.286304 2561 generic.go:356] "Generic (PLEG): container finished" podID="024f42cf-75c3-4c8d-8727-c3e07ff391a6" containerID="798f467b25d92a9577cf8416fc769ebe502f06b782ea146b71c12eb23ac5273c" exitCode=0 Jul 27 22:07:48.286466 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.286350 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" Jul 27 22:07:48.286466 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.286381 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" event={"ID":"024f42cf-75c3-4c8d-8727-c3e07ff391a6","Type":"ContainerDied","Data":"798f467b25d92a9577cf8416fc769ebe502f06b782ea146b71c12eb23ac5273c"} Jul 27 22:07:48.286466 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.286423 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz" event={"ID":"024f42cf-75c3-4c8d-8727-c3e07ff391a6","Type":"ContainerDied","Data":"40e744b1aea80b72d10597fbc032794c06584f4504a6a77ded7ddd9a0ac8a51c"} Jul 27 22:07:48.286466 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.286449 2561 scope.go:117] "RemoveContainer" containerID="798f467b25d92a9577cf8416fc769ebe502f06b782ea146b71c12eb23ac5273c" Jul 27 22:07:48.296312 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.296286 2561 scope.go:117] "RemoveContainer" containerID="798f467b25d92a9577cf8416fc769ebe502f06b782ea146b71c12eb23ac5273c" Jul 27 22:07:48.296599 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:07:48.296580 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798f467b25d92a9577cf8416fc769ebe502f06b782ea146b71c12eb23ac5273c\": container with ID starting with 798f467b25d92a9577cf8416fc769ebe502f06b782ea146b71c12eb23ac5273c not found: ID does not exist" containerID="798f467b25d92a9577cf8416fc769ebe502f06b782ea146b71c12eb23ac5273c" Jul 27 22:07:48.296672 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.296608 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798f467b25d92a9577cf8416fc769ebe502f06b782ea146b71c12eb23ac5273c"} err="failed to get container status \"798f467b25d92a9577cf8416fc769ebe502f06b782ea146b71c12eb23ac5273c\": rpc error: code = NotFound desc = could not find container \"798f467b25d92a9577cf8416fc769ebe502f06b782ea146b71c12eb23ac5273c\": container with ID starting with 798f467b25d92a9577cf8416fc769ebe502f06b782ea146b71c12eb23ac5273c not found: ID does not exist" Jul 27 22:07:48.308945 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.308920 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz"] Jul 27 22:07:48.312394 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.312368 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e7af719f-kserve-65c7789c55-jn6dz"] Jul 27 22:07:48.375074 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.375041 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-tmp-dir\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:07:48.375074 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.375069 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024f42cf-75c3-4c8d-8727-c3e07ff391a6-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:07:48.375074 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.375078 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trc6s\" (UniqueName: \"kubernetes.io/projected/024f42cf-75c3-4c8d-8727-c3e07ff391a6-kube-api-access-trc6s\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:07:48.375315 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.375089 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024f42cf-75c3-4c8d-8727-c3e07ff391a6-dshm\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:07:48.818479 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:07:48.818449 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="024f42cf-75c3-4c8d-8727-c3e07ff391a6" path="/var/lib/kubelet/pods/024f42cf-75c3-4c8d-8727-c3e07ff391a6/volumes" Jul 27 22:10:23.229470 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:10:23.229432 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:10:23.315581 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:10:23.315533 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:10:50.358618 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:10:50.358529 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:10:53.626389 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:10:53.626356 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:04.525269 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:04.525232 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:14.278551 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:14.278513 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:17.596020 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:17.595973 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:17.694179 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:17.694144 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:18.069000 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:18.068965 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:23.962380 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:23.962342 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:24.031072 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.031040 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:24.224257 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.224166 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:24.545007 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.544931 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4"] Jul 27 22:11:24.545305 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.545291 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="024f42cf-75c3-4c8d-8727-c3e07ff391a6" containerName="main" Jul 27 22:11:24.545355 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.545306 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="024f42cf-75c3-4c8d-8727-c3e07ff391a6" containerName="main" Jul 27 22:11:24.545391 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.545374 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="024f42cf-75c3-4c8d-8727-c3e07ff391a6" containerName="main" Jul 27 22:11:24.548222 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.548207 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.551449 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.551424 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a66e79c4-openshift-default-dockercfg-vwsrt\"" Jul 27 22:11:24.554176 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.554156 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-envoy\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.554295 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.554187 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-workload-certs\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.554295 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.554226 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-data\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.554384 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.554330 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istiod-ca-cert\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.554438 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.554405 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-credential-socket\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.554493 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.554470 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-token\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.554600 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.554499 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48mc\" (UniqueName: \"kubernetes.io/projected/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-kube-api-access-g48mc\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.554600 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.554532 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-podinfo\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.554600 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.554563 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-workload-socket\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.572329 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.572294 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:24.578989 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.578958 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4"] Jul 27 22:11:24.655653 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.655619 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-podinfo\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.655860 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.655664 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-workload-socket\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.655860 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.655694 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-envoy\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.655860 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.655726 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-workload-certs\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.655860 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.655779 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-data\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.655860 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.655826 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istiod-ca-cert\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.656117 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.655878 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-credential-socket\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.656117 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.655913 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-token\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.656117 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.655941 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48mc\" (UniqueName: \"kubernetes.io/projected/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-kube-api-access-g48mc\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.656279 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.656126 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-workload-socket\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.656279 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.656126 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-workload-certs\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.656279 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.656173 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-data\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.656513 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.656488 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-credential-socket\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.656855 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.656831 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istiod-ca-cert\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.658104 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.658087 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-envoy\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.658516 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.658496 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-podinfo\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.665739 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.665679 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-token\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.666454 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.666431 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48mc\" (UniqueName: \"kubernetes.io/projected/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-kube-api-access-g48mc\") pod \"e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.858820 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.858727 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:24.866666 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.866626 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:24.998300 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:24.998255 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4"] Jul 27 22:11:25.001175 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:11:25.001149 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad7ce2e_19c0_438c_b8d3_33aded1e9cfa.slice/crio-b77fabc0a0820d0385a9020ce3de4c99c6d017706a3f9701983d4db04fcbcd38 WatchSource:0}: Error finding container b77fabc0a0820d0385a9020ce3de4c99c6d017706a3f9701983d4db04fcbcd38: Status 404 returned error can't find the container with id b77fabc0a0820d0385a9020ce3de4c99c6d017706a3f9701983d4db04fcbcd38 Jul 27 22:11:25.003336 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:25.003303 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:11:25.003412 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:25.003383 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:11:25.003454 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:25.003419 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:11:25.349680 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:25.349640 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" event={"ID":"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa","Type":"ContainerStarted","Data":"2ee3ddda526514e297850b851491841c036f2f8eabc23cec24ff6d0a9901eb5b"} Jul 27 22:11:25.349883 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:25.349711 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" event={"ID":"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa","Type":"ContainerStarted","Data":"b77fabc0a0820d0385a9020ce3de4c99c6d017706a3f9701983d4db04fcbcd38"} Jul 27 22:11:25.371840 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:25.371791 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" podStartSLOduration=1.371774325 podStartE2EDuration="1.371774325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:11:25.369928919 +0000 UTC m=+1562.987366238" watchObservedRunningTime="2026-07-27 22:11:25.371774325 +0000 UTC m=+1562.989211642" Jul 27 22:11:25.858991 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:25.858936 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:26.740922 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:26.740874 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:26.834781 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:26.834734 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:26.864879 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:26.864851 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:27.012988 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:27.012884 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:27.142781 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:27.142740 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:27.359493 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:27.359397 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:27.360440 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:27.360420 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:27.599434 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:27.599396 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:27.703842 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:27.703806 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:42.361058 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:42.361022 2561 patch_prober.go:28] interesting pod/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 27 22:11:42.361525 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:42.361084 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" podUID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 27 22:11:42.438749 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:42.438718 2561 generic.go:356] "Generic (PLEG): container finished" podID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerID="2ee3ddda526514e297850b851491841c036f2f8eabc23cec24ff6d0a9901eb5b" exitCode=0 Jul 27 22:11:42.438915 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:42.438762 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" event={"ID":"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa","Type":"ContainerDied","Data":"2ee3ddda526514e297850b851491841c036f2f8eabc23cec24ff6d0a9901eb5b"} Jul 27 22:11:42.439119 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:42.439105 2561 scope.go:117] "RemoveContainer" containerID="2ee3ddda526514e297850b851491841c036f2f8eabc23cec24ff6d0a9901eb5b" Jul 27 22:11:42.440107 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:42.440078 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:11:42.440198 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:42.440173 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:11:42.440281 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:42.440225 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:11:43.452255 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:43.452218 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" event={"ID":"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa","Type":"ContainerStarted","Data":"cf9e0f0f2bbcf2c36b3965c3082677120fdd2d2c81036c5a6313b75cecd5c21e"} Jul 27 22:11:43.859833 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:43.859737 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:43.860005 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:43.859928 2561 patch_prober.go:28] interesting pod/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 27 22:11:43.860005 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:43.859972 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" podUID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 27 22:11:44.859322 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:44.859251 2561 patch_prober.go:28] interesting pod/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 27 22:11:44.859729 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:44.859355 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" podUID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 27 22:11:45.859834 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:45.859795 2561 patch_prober.go:28] interesting pod/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 27 22:11:45.860228 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:45.859868 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" podUID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 27 22:11:46.859181 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:46.859136 2561 patch_prober.go:28] interesting pod/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 27 22:11:46.859429 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:46.859198 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" podUID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 27 22:11:47.863883 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:47.863850 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:47.864397 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:47.864284 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:11:47.865622 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:11:47.865601 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:12:03.242686 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:03.242651 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:04.318437 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:04.318405 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:07.864327 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:07.864293 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:08.011423 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:08.011386 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4"] Jul 27 22:12:08.011652 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:08.011608 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" podUID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerName="istio-proxy" containerID="cri-o://cf9e0f0f2bbcf2c36b3965c3082677120fdd2d2c81036c5a6313b75cecd5c21e" gracePeriod=30 Jul 27 22:12:08.037047 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:08.037011 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:08.283050 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:08.283020 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:08.542307 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:08.542196 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:08.640169 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:08.640136 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:08.776150 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:08.776112 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:09.072866 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.072831 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms"] Jul 27 22:12:09.076333 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.076310 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.079218 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.079197 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4d2830c6-gw-openshift-default-dockercfg-q7vdg\"" Jul 27 22:12:09.090603 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.090579 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms"] Jul 27 22:12:09.111213 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.111180 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:09.259524 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.259491 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-data\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.259524 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.259525 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdba30ed-e46c-4926-94e6-9ac57fb56938-istiod-ca-cert\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.259751 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.259547 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-credential-socket\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.259751 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.259575 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-workload-certs\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.259751 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.259591 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-envoy\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.259751 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.259620 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-token\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.259751 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.259642 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zx97\" (UniqueName: \"kubernetes.io/projected/fdba30ed-e46c-4926-94e6-9ac57fb56938-kube-api-access-7zx97\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.259751 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.259683 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-workload-socket\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.259751 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.259701 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-podinfo\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.283029 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.282987 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:09.360862 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.360762 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-data\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.360862 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.360800 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdba30ed-e46c-4926-94e6-9ac57fb56938-istiod-ca-cert\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.360862 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.360824 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-credential-socket\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.361159 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.360935 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-workload-certs\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.361159 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.360962 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-envoy\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.361159 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.361004 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-token\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.361159 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.361044 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zx97\" (UniqueName: \"kubernetes.io/projected/fdba30ed-e46c-4926-94e6-9ac57fb56938-kube-api-access-7zx97\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.361159 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.361088 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-workload-socket\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.361455 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.361169 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-podinfo\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.361455 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.361293 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-data\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.361455 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.361319 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-credential-socket\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.361455 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.361404 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-workload-certs\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.361455 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.361411 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-workload-socket\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.361846 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.361827 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdba30ed-e46c-4926-94e6-9ac57fb56938-istiod-ca-cert\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.363334 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.363309 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-envoy\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.363534 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.363518 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-podinfo\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.370456 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.370429 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-token\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.370650 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.370631 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zx97\" (UniqueName: \"kubernetes.io/projected/fdba30ed-e46c-4926-94e6-9ac57fb56938-kube-api-access-7zx97\") pod \"e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.386435 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.386409 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:09.522889 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.522828 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms"] Jul 27 22:12:09.525272 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:12:09.525228 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdba30ed_e46c_4926_94e6_9ac57fb56938.slice/crio-b304812fa3f40f4c74d5e4c9e556363209c4517c506d389055cb4240377dbab2 WatchSource:0}: Error finding container b304812fa3f40f4c74d5e4c9e556363209c4517c506d389055cb4240377dbab2: Status 404 returned error can't find the container with id b304812fa3f40f4c74d5e4c9e556363209c4517c506d389055cb4240377dbab2 Jul 27 22:12:09.527603 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.527574 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:12:09.527679 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.527641 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:12:09.527679 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.527670 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:12:09.585817 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:09.585785 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" event={"ID":"fdba30ed-e46c-4926-94e6-9ac57fb56938","Type":"ContainerStarted","Data":"b304812fa3f40f4c74d5e4c9e556363209c4517c506d389055cb4240377dbab2"} Jul 27 22:12:10.592287 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:10.592223 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" event={"ID":"fdba30ed-e46c-4926-94e6-9ac57fb56938","Type":"ContainerStarted","Data":"994e111c1adebed95b3c2a1db47ef65c8b7f463465a9c8d08c5c62eee27ff8b8"} Jul 27 22:12:10.612587 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:10.612538 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" podStartSLOduration=1.612514529 podStartE2EDuration="1.612514529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:12:10.61144103 +0000 UTC m=+1608.228878370" watchObservedRunningTime="2026-07-27 22:12:10.612514529 +0000 UTC m=+1608.229951848" Jul 27 22:12:11.387678 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:11.387635 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:11.392599 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:11.392572 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:11.597159 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:11.597128 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:11.598191 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:11.598170 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:12.102044 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:12.102001 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:12.398169 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:12.398139 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:12.844472 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:12.844346 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:12.943089 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:12.943059 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:13.269232 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.269202 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:12:13.306127 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.306089 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g48mc\" (UniqueName: \"kubernetes.io/projected/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-kube-api-access-g48mc\") pod \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " Jul 27 22:12:13.306369 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.306141 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-envoy\") pod \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " Jul 27 22:12:13.306369 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.306169 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-token\") pod \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " Jul 27 22:12:13.306369 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.306333 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-podinfo\") pod \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " Jul 27 22:12:13.306554 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.306379 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-workload-socket\") pod \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " Jul 27 22:12:13.306554 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.306435 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-data\") pod \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " Jul 27 22:12:13.306554 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.306460 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istiod-ca-cert\") pod \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " Jul 27 22:12:13.306706 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.306602 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-workload-certs\") pod \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " Jul 27 22:12:13.306706 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.306629 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-credential-socket\") pod \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\" (UID: \"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa\") " Jul 27 22:12:13.306810 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.306725 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" (UID: "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:12:13.306986 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.306943 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" (UID: "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:12:13.306986 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.306978 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:13.307151 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.307131 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" (UID: "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:12:13.307666 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.307608 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-data" (OuterVolumeSpecName: "istio-data") pod "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" (UID: "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:12:13.308056 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.307804 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" (UID: "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:12:13.309184 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.309147 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" (UID: "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:12:13.309394 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.309369 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-token" (OuterVolumeSpecName: "istio-token") pod "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" (UID: "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:12:13.309713 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.309687 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" (UID: "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:12:13.309808 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.309723 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-kube-api-access-g48mc" (OuterVolumeSpecName: "kube-api-access-g48mc") pod "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" (UID: "dad7ce2e-19c0-438c-b8d3-33aded1e9cfa"). InnerVolumeSpecName "kube-api-access-g48mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:12:13.407793 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.407757 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:13.407793 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.407794 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:13.408000 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.407810 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:13.408000 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.407822 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:13.408000 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.407836 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g48mc\" (UniqueName: \"kubernetes.io/projected/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-kube-api-access-g48mc\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:13.408000 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.407848 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:13.408000 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.407860 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:13.408000 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.407872 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:13.609029 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.608992 2561 generic.go:356] "Generic (PLEG): container finished" podID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerID="cf9e0f0f2bbcf2c36b3965c3082677120fdd2d2c81036c5a6313b75cecd5c21e" exitCode=0 Jul 27 22:12:13.609240 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.609042 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" Jul 27 22:12:13.609240 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.609085 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" event={"ID":"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa","Type":"ContainerDied","Data":"cf9e0f0f2bbcf2c36b3965c3082677120fdd2d2c81036c5a6313b75cecd5c21e"} Jul 27 22:12:13.609240 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.609138 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4" event={"ID":"dad7ce2e-19c0-438c-b8d3-33aded1e9cfa","Type":"ContainerDied","Data":"b77fabc0a0820d0385a9020ce3de4c99c6d017706a3f9701983d4db04fcbcd38"} Jul 27 22:12:13.609240 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.609160 2561 scope.go:117] "RemoveContainer" containerID="cf9e0f0f2bbcf2c36b3965c3082677120fdd2d2c81036c5a6313b75cecd5c21e" Jul 27 22:12:13.618919 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.618898 2561 scope.go:117] "RemoveContainer" containerID="2ee3ddda526514e297850b851491841c036f2f8eabc23cec24ff6d0a9901eb5b" Jul 27 22:12:13.627531 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.627514 2561 scope.go:117] "RemoveContainer" containerID="cf9e0f0f2bbcf2c36b3965c3082677120fdd2d2c81036c5a6313b75cecd5c21e" Jul 27 22:12:13.627903 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:12:13.627818 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf9e0f0f2bbcf2c36b3965c3082677120fdd2d2c81036c5a6313b75cecd5c21e\": container with ID starting with cf9e0f0f2bbcf2c36b3965c3082677120fdd2d2c81036c5a6313b75cecd5c21e not found: ID does not exist" containerID="cf9e0f0f2bbcf2c36b3965c3082677120fdd2d2c81036c5a6313b75cecd5c21e" Jul 27 22:12:13.627903 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.627862 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9e0f0f2bbcf2c36b3965c3082677120fdd2d2c81036c5a6313b75cecd5c21e"} err="failed to get container status \"cf9e0f0f2bbcf2c36b3965c3082677120fdd2d2c81036c5a6313b75cecd5c21e\": rpc error: code = NotFound desc = could not find container \"cf9e0f0f2bbcf2c36b3965c3082677120fdd2d2c81036c5a6313b75cecd5c21e\": container with ID starting with cf9e0f0f2bbcf2c36b3965c3082677120fdd2d2c81036c5a6313b75cecd5c21e not found: ID does not exist" Jul 27 22:12:13.627903 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.627889 2561 scope.go:117] "RemoveContainer" containerID="2ee3ddda526514e297850b851491841c036f2f8eabc23cec24ff6d0a9901eb5b" Jul 27 22:12:13.628216 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:12:13.628195 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee3ddda526514e297850b851491841c036f2f8eabc23cec24ff6d0a9901eb5b\": container with ID starting with 2ee3ddda526514e297850b851491841c036f2f8eabc23cec24ff6d0a9901eb5b not found: ID does not exist" containerID="2ee3ddda526514e297850b851491841c036f2f8eabc23cec24ff6d0a9901eb5b" Jul 27 22:12:13.628334 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.628224 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee3ddda526514e297850b851491841c036f2f8eabc23cec24ff6d0a9901eb5b"} err="failed to get container status \"2ee3ddda526514e297850b851491841c036f2f8eabc23cec24ff6d0a9901eb5b\": rpc error: code = NotFound desc = could not find container \"2ee3ddda526514e297850b851491841c036f2f8eabc23cec24ff6d0a9901eb5b\": container with ID starting with 2ee3ddda526514e297850b851491841c036f2f8eabc23cec24ff6d0a9901eb5b not found: ID does not exist" Jul 27 22:12:13.630591 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.630572 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4"] Jul 27 22:12:13.634553 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:13.634532 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a66e79c4-openshift-default-db5dcbc86-d62r4"] Jul 27 22:12:14.818448 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:14.818349 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" path="/var/lib/kubelet/pods/dad7ce2e-19c0-438c-b8d3-33aded1e9cfa/volumes" Jul 27 22:12:25.681322 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:25.681252 2561 generic.go:356] "Generic (PLEG): container finished" podID="fdba30ed-e46c-4926-94e6-9ac57fb56938" containerID="994e111c1adebed95b3c2a1db47ef65c8b7f463465a9c8d08c5c62eee27ff8b8" exitCode=0 Jul 27 22:12:25.681771 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:25.681327 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" event={"ID":"fdba30ed-e46c-4926-94e6-9ac57fb56938","Type":"ContainerDied","Data":"994e111c1adebed95b3c2a1db47ef65c8b7f463465a9c8d08c5c62eee27ff8b8"} Jul 27 22:12:25.681840 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:25.681824 2561 scope.go:117] "RemoveContainer" containerID="994e111c1adebed95b3c2a1db47ef65c8b7f463465a9c8d08c5c62eee27ff8b8" Jul 27 22:12:25.682956 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:25.682932 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:12:25.683030 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:25.682992 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:12:25.683030 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:25.683023 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:12:26.597449 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:26.597392 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:26.688736 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:26.688699 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" event={"ID":"fdba30ed-e46c-4926-94e6-9ac57fb56938","Type":"ContainerStarted","Data":"b79d23fc06d5da7e946df48da6a53b93a0116f7b7309e326364868466499b434"} Jul 27 22:12:27.387460 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:27.387420 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:27.388719 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:27.388693 2561 patch_prober.go:28] interesting pod/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 27 22:12:27.388841 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:27.388745 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" podUID="fdba30ed-e46c-4926-94e6-9ac57fb56938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 27 22:12:28.387286 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:28.387228 2561 patch_prober.go:28] interesting pod/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 27 22:12:28.387708 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:28.387319 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" podUID="fdba30ed-e46c-4926-94e6-9ac57fb56938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 27 22:12:29.387770 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:29.387729 2561 patch_prober.go:28] interesting pod/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 27 22:12:29.388184 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:29.387790 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" podUID="fdba30ed-e46c-4926-94e6-9ac57fb56938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 27 22:12:30.387175 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:30.387131 2561 patch_prober.go:28] interesting pod/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 27 22:12:30.387400 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:30.387234 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" podUID="fdba30ed-e46c-4926-94e6-9ac57fb56938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 27 22:12:31.391549 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:31.391518 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:31.392002 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:31.391813 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:31.392601 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:31.392582 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:47.668908 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:47.668870 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:47.745318 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:47.745280 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:47.810208 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:47.810171 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:52.272716 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:52.272682 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:52.423538 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:52.423502 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms"] Jul 27 22:12:52.423799 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:52.423768 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" podUID="fdba30ed-e46c-4926-94e6-9ac57fb56938" containerName="istio-proxy" containerID="cri-o://b79d23fc06d5da7e946df48da6a53b93a0116f7b7309e326364868466499b434" gracePeriod=30 Jul 27 22:12:52.446445 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:52.446408 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:52.746100 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:52.746070 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:52.953284 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:52.953224 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:53.295965 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.295930 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6"] Jul 27 22:12:53.296471 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.296417 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerName="istio-proxy" Jul 27 22:12:53.296471 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.296436 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerName="istio-proxy" Jul 27 22:12:53.296593 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.296563 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerName="istio-proxy" Jul 27 22:12:53.298522 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.298502 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.301139 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.301121 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d4157b7d-openshift-default-dockercfg-rmfkw\"" Jul 27 22:12:53.308239 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.308213 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:53.318435 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.318405 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6"] Jul 27 22:12:53.394529 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.394494 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0589f69d-01df-4378-a2a3-75df57c3d558-istio-token\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.394529 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.394530 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0589f69d-01df-4378-a2a3-75df57c3d558-istiod-ca-cert\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.394760 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.394568 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkfj4\" (UniqueName: \"kubernetes.io/projected/0589f69d-01df-4378-a2a3-75df57c3d558-kube-api-access-rkfj4\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.394760 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.394686 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-workload-socket\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.394760 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.394742 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-credential-socket\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.394871 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.394767 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-istio-data\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.394871 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.394801 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-istio-envoy\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.394871 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.394834 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-workload-certs\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.394968 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.394898 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0589f69d-01df-4378-a2a3-75df57c3d558-istio-podinfo\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.495364 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.495330 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-istio-envoy\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.495364 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.495364 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-workload-certs\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.495630 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.495405 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0589f69d-01df-4378-a2a3-75df57c3d558-istio-podinfo\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.495630 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.495517 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0589f69d-01df-4378-a2a3-75df57c3d558-istio-token\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.495630 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.495578 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0589f69d-01df-4378-a2a3-75df57c3d558-istiod-ca-cert\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.495784 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.495632 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkfj4\" (UniqueName: \"kubernetes.io/projected/0589f69d-01df-4378-a2a3-75df57c3d558-kube-api-access-rkfj4\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.495784 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.495699 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-workload-socket\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.495784 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.495741 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-credential-socket\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.495784 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.495770 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-istio-data\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.496009 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.495811 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-workload-certs\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.496009 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.495991 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-workload-socket\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.496121 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.496097 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-credential-socket\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.496206 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.496187 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-istio-data\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.496318 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.496301 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0589f69d-01df-4378-a2a3-75df57c3d558-istiod-ca-cert\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.497869 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.497851 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0589f69d-01df-4378-a2a3-75df57c3d558-istio-podinfo\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.497933 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.497904 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-istio-envoy\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.503593 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.503569 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0589f69d-01df-4378-a2a3-75df57c3d558-istio-token\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.503787 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.503764 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkfj4\" (UniqueName: \"kubernetes.io/projected/0589f69d-01df-4378-a2a3-75df57c3d558-kube-api-access-rkfj4\") pod \"e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.535055 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.534994 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:53.610667 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.610566 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:53.750381 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.750352 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6"] Jul 27 22:12:53.752324 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:12:53.752294 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0589f69d_01df_4378_a2a3_75df57c3d558.slice/crio-d12e3f6006925cc987a0af9d43290435c6d6dcc055044c70623eb5e0bda56668 WatchSource:0}: Error finding container d12e3f6006925cc987a0af9d43290435c6d6dcc055044c70623eb5e0bda56668: Status 404 returned error can't find the container with id d12e3f6006925cc987a0af9d43290435c6d6dcc055044c70623eb5e0bda56668 Jul 27 22:12:53.754602 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.754571 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:12:53.754684 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.754633 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:12:53.754684 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.754660 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:12:53.833497 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.833462 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" event={"ID":"0589f69d-01df-4378-a2a3-75df57c3d558","Type":"ContainerStarted","Data":"e8a097bf404f392bb8a6b7304330dadb2ccaf9a118957e9c39b338d812b320f4"} Jul 27 22:12:53.833686 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.833550 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" event={"ID":"0589f69d-01df-4378-a2a3-75df57c3d558","Type":"ContainerStarted","Data":"d12e3f6006925cc987a0af9d43290435c6d6dcc055044c70623eb5e0bda56668"} Jul 27 22:12:53.855024 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:53.854973 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" podStartSLOduration=0.854956562 podStartE2EDuration="854.956562ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:12:53.853312652 +0000 UTC m=+1651.470749973" watchObservedRunningTime="2026-07-27 22:12:53.854956562 +0000 UTC m=+1651.472393882" Jul 27 22:12:54.610994 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:54.610961 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:54.615819 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:54.615794 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:54.838929 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:54.838905 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:54.839950 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:54.839929 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:12:55.227891 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:55.227859 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:55.327726 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:55.327688 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:55.529895 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:55.529793 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:55.837904 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:55.837798 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:56.255291 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:56.255237 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:56.349155 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:56.349098 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:57.670792 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.670768 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:57.736501 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.736413 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-envoy\") pod \"fdba30ed-e46c-4926-94e6-9ac57fb56938\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " Jul 27 22:12:57.736501 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.736452 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zx97\" (UniqueName: \"kubernetes.io/projected/fdba30ed-e46c-4926-94e6-9ac57fb56938-kube-api-access-7zx97\") pod \"fdba30ed-e46c-4926-94e6-9ac57fb56938\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " Jul 27 22:12:57.736501 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.736470 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-workload-socket\") pod \"fdba30ed-e46c-4926-94e6-9ac57fb56938\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " Jul 27 22:12:57.736788 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.736515 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-data\") pod \"fdba30ed-e46c-4926-94e6-9ac57fb56938\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " Jul 27 22:12:57.736788 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.736570 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdba30ed-e46c-4926-94e6-9ac57fb56938-istiod-ca-cert\") pod \"fdba30ed-e46c-4926-94e6-9ac57fb56938\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " Jul 27 22:12:57.736788 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.736595 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-token\") pod \"fdba30ed-e46c-4926-94e6-9ac57fb56938\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " Jul 27 22:12:57.736788 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.736647 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-podinfo\") pod \"fdba30ed-e46c-4926-94e6-9ac57fb56938\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " Jul 27 22:12:57.736788 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.736671 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-credential-socket\") pod \"fdba30ed-e46c-4926-94e6-9ac57fb56938\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " Jul 27 22:12:57.736788 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.736693 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-workload-certs\") pod \"fdba30ed-e46c-4926-94e6-9ac57fb56938\" (UID: \"fdba30ed-e46c-4926-94e6-9ac57fb56938\") " Jul 27 22:12:57.737090 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.736866 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fdba30ed-e46c-4926-94e6-9ac57fb56938" (UID: "fdba30ed-e46c-4926-94e6-9ac57fb56938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:12:57.737090 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.736979 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdba30ed-e46c-4926-94e6-9ac57fb56938-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fdba30ed-e46c-4926-94e6-9ac57fb56938" (UID: "fdba30ed-e46c-4926-94e6-9ac57fb56938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:12:57.737090 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.737048 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:57.737090 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.737060 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fdba30ed-e46c-4926-94e6-9ac57fb56938" (UID: "fdba30ed-e46c-4926-94e6-9ac57fb56938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:12:57.737322 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.737247 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fdba30ed-e46c-4926-94e6-9ac57fb56938" (UID: "fdba30ed-e46c-4926-94e6-9ac57fb56938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:12:57.737456 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.737435 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-data" (OuterVolumeSpecName: "istio-data") pod "fdba30ed-e46c-4926-94e6-9ac57fb56938" (UID: "fdba30ed-e46c-4926-94e6-9ac57fb56938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:12:57.738844 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.738815 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-token" (OuterVolumeSpecName: "istio-token") pod "fdba30ed-e46c-4926-94e6-9ac57fb56938" (UID: "fdba30ed-e46c-4926-94e6-9ac57fb56938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:12:57.738938 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.738909 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fdba30ed-e46c-4926-94e6-9ac57fb56938" (UID: "fdba30ed-e46c-4926-94e6-9ac57fb56938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:12:57.739171 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.739147 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fdba30ed-e46c-4926-94e6-9ac57fb56938" (UID: "fdba30ed-e46c-4926-94e6-9ac57fb56938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:12:57.739296 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.739168 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdba30ed-e46c-4926-94e6-9ac57fb56938-kube-api-access-7zx97" (OuterVolumeSpecName: "kube-api-access-7zx97") pod "fdba30ed-e46c-4926-94e6-9ac57fb56938" (UID: "fdba30ed-e46c-4926-94e6-9ac57fb56938"). InnerVolumeSpecName "kube-api-access-7zx97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:12:57.838456 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.838420 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:57.838456 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.838451 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:57.838456 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.838461 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:57.838708 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.838471 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:57.838708 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.838480 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zx97\" (UniqueName: \"kubernetes.io/projected/fdba30ed-e46c-4926-94e6-9ac57fb56938-kube-api-access-7zx97\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:57.838708 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.838488 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:57.838708 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.838496 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdba30ed-e46c-4926-94e6-9ac57fb56938-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:57.838708 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.838505 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdba30ed-e46c-4926-94e6-9ac57fb56938-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:12:57.855405 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.855370 2561 generic.go:356] "Generic (PLEG): container finished" podID="fdba30ed-e46c-4926-94e6-9ac57fb56938" containerID="b79d23fc06d5da7e946df48da6a53b93a0116f7b7309e326364868466499b434" exitCode=0 Jul 27 22:12:57.855587 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.855418 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" Jul 27 22:12:57.855587 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.855457 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" event={"ID":"fdba30ed-e46c-4926-94e6-9ac57fb56938","Type":"ContainerDied","Data":"b79d23fc06d5da7e946df48da6a53b93a0116f7b7309e326364868466499b434"} Jul 27 22:12:57.855587 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.855511 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms" event={"ID":"fdba30ed-e46c-4926-94e6-9ac57fb56938","Type":"ContainerDied","Data":"b304812fa3f40f4c74d5e4c9e556363209c4517c506d389055cb4240377dbab2"} Jul 27 22:12:57.855587 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.855536 2561 scope.go:117] "RemoveContainer" containerID="b79d23fc06d5da7e946df48da6a53b93a0116f7b7309e326364868466499b434" Jul 27 22:12:57.865103 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.865077 2561 scope.go:117] "RemoveContainer" containerID="994e111c1adebed95b3c2a1db47ef65c8b7f463465a9c8d08c5c62eee27ff8b8" Jul 27 22:12:57.873825 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.873805 2561 scope.go:117] "RemoveContainer" containerID="b79d23fc06d5da7e946df48da6a53b93a0116f7b7309e326364868466499b434" Jul 27 22:12:57.874105 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:12:57.874086 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79d23fc06d5da7e946df48da6a53b93a0116f7b7309e326364868466499b434\": container with ID starting with b79d23fc06d5da7e946df48da6a53b93a0116f7b7309e326364868466499b434 not found: ID does not exist" containerID="b79d23fc06d5da7e946df48da6a53b93a0116f7b7309e326364868466499b434" Jul 27 22:12:57.874154 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.874115 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79d23fc06d5da7e946df48da6a53b93a0116f7b7309e326364868466499b434"} err="failed to get container status \"b79d23fc06d5da7e946df48da6a53b93a0116f7b7309e326364868466499b434\": rpc error: code = NotFound desc = could not find container \"b79d23fc06d5da7e946df48da6a53b93a0116f7b7309e326364868466499b434\": container with ID starting with b79d23fc06d5da7e946df48da6a53b93a0116f7b7309e326364868466499b434 not found: ID does not exist" Jul 27 22:12:57.874154 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.874133 2561 scope.go:117] "RemoveContainer" containerID="994e111c1adebed95b3c2a1db47ef65c8b7f463465a9c8d08c5c62eee27ff8b8" Jul 27 22:12:57.874434 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:12:57.874414 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"994e111c1adebed95b3c2a1db47ef65c8b7f463465a9c8d08c5c62eee27ff8b8\": container with ID starting with 994e111c1adebed95b3c2a1db47ef65c8b7f463465a9c8d08c5c62eee27ff8b8 not found: ID does not exist" containerID="994e111c1adebed95b3c2a1db47ef65c8b7f463465a9c8d08c5c62eee27ff8b8" Jul 27 22:12:57.874507 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.874439 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"994e111c1adebed95b3c2a1db47ef65c8b7f463465a9c8d08c5c62eee27ff8b8"} err="failed to get container status \"994e111c1adebed95b3c2a1db47ef65c8b7f463465a9c8d08c5c62eee27ff8b8\": rpc error: code = NotFound desc = could not find container \"994e111c1adebed95b3c2a1db47ef65c8b7f463465a9c8d08c5c62eee27ff8b8\": container with ID starting with 994e111c1adebed95b3c2a1db47ef65c8b7f463465a9c8d08c5c62eee27ff8b8 not found: ID does not exist" Jul 27 22:12:57.887222 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.887194 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms"] Jul 27 22:12:57.892886 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:57.892861 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4d2830c6-gw-openshift-default-578c9b744d-nbjms"] Jul 27 22:12:58.818078 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:12:58.818041 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdba30ed-e46c-4926-94e6-9ac57fb56938" path="/var/lib/kubelet/pods/fdba30ed-e46c-4926-94e6-9ac57fb56938/volumes" Jul 27 22:13:09.841005 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:09.840920 2561 patch_prober.go:28] interesting pod/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 27 22:13:09.841427 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:09.841000 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" podUID="0589f69d-01df-4378-a2a3-75df57c3d558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 27 22:13:09.917823 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:09.917790 2561 generic.go:356] "Generic (PLEG): container finished" podID="0589f69d-01df-4378-a2a3-75df57c3d558" containerID="e8a097bf404f392bb8a6b7304330dadb2ccaf9a118957e9c39b338d812b320f4" exitCode=0 Jul 27 22:13:09.917993 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:09.917852 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" event={"ID":"0589f69d-01df-4378-a2a3-75df57c3d558","Type":"ContainerDied","Data":"e8a097bf404f392bb8a6b7304330dadb2ccaf9a118957e9c39b338d812b320f4"} Jul 27 22:13:09.918312 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:09.918295 2561 scope.go:117] "RemoveContainer" containerID="e8a097bf404f392bb8a6b7304330dadb2ccaf9a118957e9c39b338d812b320f4" Jul 27 22:13:09.919317 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:09.919289 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:13:09.919403 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:09.919360 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:13:09.919403 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:09.919394 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:13:10.924743 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:10.924709 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" event={"ID":"0589f69d-01df-4378-a2a3-75df57c3d558","Type":"ContainerStarted","Data":"51e8e61d6aa2fbdfe2e3f0d240ddd07066497868fc2aa70c1a054996f049722a"} Jul 27 22:13:11.611162 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:11.611123 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:13:11.611428 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:11.611406 2561 patch_prober.go:28] interesting pod/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 27 22:13:11.611531 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:11.611452 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" podUID="0589f69d-01df-4378-a2a3-75df57c3d558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 27 22:13:12.612022 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:12.611981 2561 patch_prober.go:28] interesting pod/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 27 22:13:12.612468 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:12.612041 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" podUID="0589f69d-01df-4378-a2a3-75df57c3d558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 27 22:13:13.611851 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:13.611813 2561 patch_prober.go:28] interesting pod/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 27 22:13:13.612071 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:13.611874 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" podUID="0589f69d-01df-4378-a2a3-75df57c3d558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 27 22:13:14.611284 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:14.611230 2561 patch_prober.go:28] interesting pod/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 27 22:13:14.611475 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:14.611316 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" podUID="0589f69d-01df-4378-a2a3-75df57c3d558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 27 22:13:15.615871 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:15.615834 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:13:15.616280 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:15.616118 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:13:15.616945 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:15.616927 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:13:31.615583 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:31.615527 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:31.694408 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:31.694360 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:31.768832 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:31.768795 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:36.267956 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:36.267918 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:37.508811 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:37.508773 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6"] Jul 27 22:13:37.509246 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:37.509029 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" podUID="0589f69d-01df-4378-a2a3-75df57c3d558" containerName="istio-proxy" containerID="cri-o://51e8e61d6aa2fbdfe2e3f0d240ddd07066497868fc2aa70c1a054996f049722a" gracePeriod=30 Jul 27 22:13:37.529928 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:37.529893 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:37.772329 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:37.772226 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:38.032589 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.032491 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:38.328600 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.328516 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4"] Jul 27 22:13:38.328876 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.328863 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdba30ed-e46c-4926-94e6-9ac57fb56938" containerName="istio-proxy" Jul 27 22:13:38.328924 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.328878 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdba30ed-e46c-4926-94e6-9ac57fb56938" containerName="istio-proxy" Jul 27 22:13:38.328924 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.328899 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerName="istio-proxy" Jul 27 22:13:38.328924 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.328905 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerName="istio-proxy" Jul 27 22:13:38.328924 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.328910 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdba30ed-e46c-4926-94e6-9ac57fb56938" containerName="istio-proxy" Jul 27 22:13:38.328924 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.328916 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdba30ed-e46c-4926-94e6-9ac57fb56938" containerName="istio-proxy" Jul 27 22:13:38.329072 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.328965 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdba30ed-e46c-4926-94e6-9ac57fb56938" containerName="istio-proxy" Jul 27 22:13:38.329072 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.328974 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdba30ed-e46c-4926-94e6-9ac57fb56938" containerName="istio-proxy" Jul 27 22:13:38.329072 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.328980 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad7ce2e-19c0-438c-b8d3-33aded1e9cfa" containerName="istio-proxy" Jul 27 22:13:38.331869 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.331853 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.334125 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.334103 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-78fe627e-openshift-default-dockercfg-j9dgm\"" Jul 27 22:13:38.344502 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.344478 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4"] Jul 27 22:13:38.353343 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.353311 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:38.494607 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.494567 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-workload-socket\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.494607 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.494608 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-data\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.494850 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.494641 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-envoy\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.494850 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.494705 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istiod-ca-cert\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.494850 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.494744 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-credential-socket\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.494850 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.494781 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5qnf\" (UniqueName: \"kubernetes.io/projected/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-kube-api-access-w5qnf\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.494850 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.494830 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-workload-certs\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.494850 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.494847 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-podinfo\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.495116 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.494873 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-token\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.595977 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.595889 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-envoy\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.595977 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.595929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istiod-ca-cert\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.595977 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.595957 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-credential-socket\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.596604 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.596005 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5qnf\" (UniqueName: \"kubernetes.io/projected/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-kube-api-access-w5qnf\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.596604 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.596038 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-workload-certs\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.596604 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.596107 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-podinfo\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.596604 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.596168 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-token\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.596604 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.596281 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-workload-socket\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.596604 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.596308 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-data\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.596604 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.596516 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-workload-certs\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.596974 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.596829 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-data\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.596974 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.596845 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-workload-socket\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.596974 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.596836 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istiod-ca-cert\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.596974 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.596901 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-credential-socket\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.598391 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.598370 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-envoy\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.598737 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.598715 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-podinfo\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.605066 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.605041 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5qnf\" (UniqueName: \"kubernetes.io/projected/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-kube-api-access-w5qnf\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.605629 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.605604 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-token\") pod \"e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.642963 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.642936 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:38.789186 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.789156 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4"] Jul 27 22:13:38.800865 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.800830 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:13:38.800967 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.800900 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:13:38.800967 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.800927 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:13:38.881488 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:38.881453 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:39.082746 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:39.082706 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" event={"ID":"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7","Type":"ContainerStarted","Data":"b74f919c4476298400c85b11dd56e0a7f28b1eef34794b390e263d2700be3ffe"} Jul 27 22:13:39.082927 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:39.082771 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" event={"ID":"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7","Type":"ContainerStarted","Data":"062352628a491be0a755d9448a271e919c580a0dedc6b42fe55e6fd0afd225f3"} Jul 27 22:13:39.104010 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:39.103958 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" podStartSLOduration=1.10394403 podStartE2EDuration="1.10394403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:13:39.102563267 +0000 UTC m=+1696.720000587" watchObservedRunningTime="2026-07-27 22:13:39.10394403 +0000 UTC m=+1696.721381350" Jul 27 22:13:39.643421 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:39.643386 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:39.648342 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:39.648316 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:40.088136 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:40.088108 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:40.089028 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:40.089006 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:40.446789 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:40.446750 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:40.545293 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:40.545245 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:40.733817 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:40.733722 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:41.073747 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:41.073651 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:41.360857 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:41.360761 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:41.458770 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:41.458734 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:42.766157 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.766132 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:13:42.829168 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.829085 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0589f69d-01df-4378-a2a3-75df57c3d558-istio-token\") pod \"0589f69d-01df-4378-a2a3-75df57c3d558\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " Jul 27 22:13:42.829168 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.829132 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-workload-socket\") pod \"0589f69d-01df-4378-a2a3-75df57c3d558\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " Jul 27 22:13:42.829168 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.829159 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-istio-envoy\") pod \"0589f69d-01df-4378-a2a3-75df57c3d558\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " Jul 27 22:13:42.829466 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.829192 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-istio-data\") pod \"0589f69d-01df-4378-a2a3-75df57c3d558\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " Jul 27 22:13:42.829466 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.829229 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkfj4\" (UniqueName: \"kubernetes.io/projected/0589f69d-01df-4378-a2a3-75df57c3d558-kube-api-access-rkfj4\") pod \"0589f69d-01df-4378-a2a3-75df57c3d558\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " Jul 27 22:13:42.829466 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.829252 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0589f69d-01df-4378-a2a3-75df57c3d558-istio-podinfo\") pod \"0589f69d-01df-4378-a2a3-75df57c3d558\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " Jul 27 22:13:42.829466 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.829340 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0589f69d-01df-4378-a2a3-75df57c3d558-istiod-ca-cert\") pod \"0589f69d-01df-4378-a2a3-75df57c3d558\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " Jul 27 22:13:42.829466 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.829367 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-credential-socket\") pod \"0589f69d-01df-4378-a2a3-75df57c3d558\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " Jul 27 22:13:42.829466 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.829392 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-workload-certs\") pod \"0589f69d-01df-4378-a2a3-75df57c3d558\" (UID: \"0589f69d-01df-4378-a2a3-75df57c3d558\") " Jul 27 22:13:42.829787 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.829487 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0589f69d-01df-4378-a2a3-75df57c3d558" (UID: "0589f69d-01df-4378-a2a3-75df57c3d558"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:13:42.829787 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.829698 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0589f69d-01df-4378-a2a3-75df57c3d558" (UID: "0589f69d-01df-4378-a2a3-75df57c3d558"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:13:42.829904 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.829779 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0589f69d-01df-4378-a2a3-75df57c3d558-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0589f69d-01df-4378-a2a3-75df57c3d558" (UID: "0589f69d-01df-4378-a2a3-75df57c3d558"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:13:42.829967 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.829901 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0589f69d-01df-4378-a2a3-75df57c3d558" (UID: "0589f69d-01df-4378-a2a3-75df57c3d558"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:13:42.830208 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.830188 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:13:42.830353 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.830236 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-istio-data" (OuterVolumeSpecName: "istio-data") pod "0589f69d-01df-4378-a2a3-75df57c3d558" (UID: "0589f69d-01df-4378-a2a3-75df57c3d558"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:13:42.831506 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.831476 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0589f69d-01df-4378-a2a3-75df57c3d558" (UID: "0589f69d-01df-4378-a2a3-75df57c3d558"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:13:42.831616 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.831586 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0589f69d-01df-4378-a2a3-75df57c3d558-istio-token" (OuterVolumeSpecName: "istio-token") pod "0589f69d-01df-4378-a2a3-75df57c3d558" (UID: "0589f69d-01df-4378-a2a3-75df57c3d558"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:13:42.831871 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.831851 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0589f69d-01df-4378-a2a3-75df57c3d558-kube-api-access-rkfj4" (OuterVolumeSpecName: "kube-api-access-rkfj4") pod "0589f69d-01df-4378-a2a3-75df57c3d558" (UID: "0589f69d-01df-4378-a2a3-75df57c3d558"). InnerVolumeSpecName "kube-api-access-rkfj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:13:42.832124 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.832110 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0589f69d-01df-4378-a2a3-75df57c3d558-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0589f69d-01df-4378-a2a3-75df57c3d558" (UID: "0589f69d-01df-4378-a2a3-75df57c3d558"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:13:42.931201 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.931172 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0589f69d-01df-4378-a2a3-75df57c3d558-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:13:42.931201 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.931198 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:13:42.931201 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.931215 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:13:42.931488 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.931228 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0589f69d-01df-4378-a2a3-75df57c3d558-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:13:42.931488 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.931239 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:13:42.931488 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.931249 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0589f69d-01df-4378-a2a3-75df57c3d558-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:13:42.931488 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.931272 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkfj4\" (UniqueName: \"kubernetes.io/projected/0589f69d-01df-4378-a2a3-75df57c3d558-kube-api-access-rkfj4\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:13:42.931488 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:42.931284 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0589f69d-01df-4378-a2a3-75df57c3d558-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:13:43.104893 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:43.104790 2561 generic.go:356] "Generic (PLEG): container finished" podID="0589f69d-01df-4378-a2a3-75df57c3d558" containerID="51e8e61d6aa2fbdfe2e3f0d240ddd07066497868fc2aa70c1a054996f049722a" exitCode=0 Jul 27 22:13:43.104893 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:43.104847 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" Jul 27 22:13:43.104893 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:43.104875 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" event={"ID":"0589f69d-01df-4378-a2a3-75df57c3d558","Type":"ContainerDied","Data":"51e8e61d6aa2fbdfe2e3f0d240ddd07066497868fc2aa70c1a054996f049722a"} Jul 27 22:13:43.105172 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:43.104939 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6" event={"ID":"0589f69d-01df-4378-a2a3-75df57c3d558","Type":"ContainerDied","Data":"d12e3f6006925cc987a0af9d43290435c6d6dcc055044c70623eb5e0bda56668"} Jul 27 22:13:43.105172 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:43.104969 2561 scope.go:117] "RemoveContainer" containerID="51e8e61d6aa2fbdfe2e3f0d240ddd07066497868fc2aa70c1a054996f049722a" Jul 27 22:13:43.114581 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:43.114560 2561 scope.go:117] "RemoveContainer" containerID="e8a097bf404f392bb8a6b7304330dadb2ccaf9a118957e9c39b338d812b320f4" Jul 27 22:13:43.123225 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:43.123205 2561 scope.go:117] "RemoveContainer" containerID="51e8e61d6aa2fbdfe2e3f0d240ddd07066497868fc2aa70c1a054996f049722a" Jul 27 22:13:43.123515 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:13:43.123491 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e8e61d6aa2fbdfe2e3f0d240ddd07066497868fc2aa70c1a054996f049722a\": container with ID starting with 51e8e61d6aa2fbdfe2e3f0d240ddd07066497868fc2aa70c1a054996f049722a not found: ID does not exist" containerID="51e8e61d6aa2fbdfe2e3f0d240ddd07066497868fc2aa70c1a054996f049722a" Jul 27 22:13:43.123576 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:43.123524 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e8e61d6aa2fbdfe2e3f0d240ddd07066497868fc2aa70c1a054996f049722a"} err="failed to get container status \"51e8e61d6aa2fbdfe2e3f0d240ddd07066497868fc2aa70c1a054996f049722a\": rpc error: code = NotFound desc = could not find container \"51e8e61d6aa2fbdfe2e3f0d240ddd07066497868fc2aa70c1a054996f049722a\": container with ID starting with 51e8e61d6aa2fbdfe2e3f0d240ddd07066497868fc2aa70c1a054996f049722a not found: ID does not exist" Jul 27 22:13:43.123576 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:43.123541 2561 scope.go:117] "RemoveContainer" containerID="e8a097bf404f392bb8a6b7304330dadb2ccaf9a118957e9c39b338d812b320f4" Jul 27 22:13:43.123770 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:13:43.123755 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a097bf404f392bb8a6b7304330dadb2ccaf9a118957e9c39b338d812b320f4\": container with ID starting with e8a097bf404f392bb8a6b7304330dadb2ccaf9a118957e9c39b338d812b320f4 not found: ID does not exist" containerID="e8a097bf404f392bb8a6b7304330dadb2ccaf9a118957e9c39b338d812b320f4" Jul 27 22:13:43.123815 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:43.123774 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a097bf404f392bb8a6b7304330dadb2ccaf9a118957e9c39b338d812b320f4"} err="failed to get container status \"e8a097bf404f392bb8a6b7304330dadb2ccaf9a118957e9c39b338d812b320f4\": rpc error: code = NotFound desc = could not find container \"e8a097bf404f392bb8a6b7304330dadb2ccaf9a118957e9c39b338d812b320f4\": container with ID starting with e8a097bf404f392bb8a6b7304330dadb2ccaf9a118957e9c39b338d812b320f4 not found: ID does not exist" Jul 27 22:13:43.132985 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:43.132961 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6"] Jul 27 22:13:43.138437 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:43.138414 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d4157b7d-openshift-default-6dbc768884-9jfl6"] Jul 27 22:13:44.818505 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:44.818427 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0589f69d-01df-4378-a2a3-75df57c3d558" path="/var/lib/kubelet/pods/0589f69d-01df-4378-a2a3-75df57c3d558/volumes" Jul 27 22:13:54.174555 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:54.174470 2561 generic.go:356] "Generic (PLEG): container finished" podID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" containerID="b74f919c4476298400c85b11dd56e0a7f28b1eef34794b390e263d2700be3ffe" exitCode=0 Jul 27 22:13:54.175071 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:54.174548 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" event={"ID":"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7","Type":"ContainerDied","Data":"b74f919c4476298400c85b11dd56e0a7f28b1eef34794b390e263d2700be3ffe"} Jul 27 22:13:54.175071 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:54.175063 2561 scope.go:117] "RemoveContainer" containerID="b74f919c4476298400c85b11dd56e0a7f28b1eef34794b390e263d2700be3ffe" Jul 27 22:13:54.176177 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:54.176151 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:13:54.176246 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:54.176234 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:13:54.176320 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:54.176306 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:13:55.088435 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:55.088389 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:55.181650 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:55.181618 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" event={"ID":"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7","Type":"ContainerStarted","Data":"7d721813a44569f53097f8274b149682f481add7102e1c3aa9d2146e39cc013a"} Jul 27 22:13:55.644987 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:55.644944 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:55.645212 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:55.644997 2561 patch_prober.go:28] interesting pod/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 27 22:13:55.645212 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:55.645078 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" podUID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 27 22:13:56.643810 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:56.643771 2561 patch_prober.go:28] interesting pod/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 27 22:13:56.644463 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:56.644427 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" podUID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 27 22:13:57.643581 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:57.643530 2561 patch_prober.go:28] interesting pod/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 27 22:13:57.643816 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:57.643612 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" podUID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 27 22:13:58.644358 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:58.644312 2561 patch_prober.go:28] interesting pod/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 27 22:13:58.644886 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:58.644382 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" podUID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 27 22:13:59.648004 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:59.647962 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:59.648397 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:59.648349 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:13:59.648988 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:13:59.648969 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:14:16.011574 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:16.011535 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:16.086763 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:16.086729 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:16.146951 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:16.146918 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:20.478756 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:20.478696 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:20.619554 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:20.619519 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4"] Jul 27 22:14:20.619773 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:20.619746 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" podUID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" containerName="istio-proxy" containerID="cri-o://7d721813a44569f53097f8274b149682f481add7102e1c3aa9d2146e39cc013a" gracePeriod=30 Jul 27 22:14:20.640523 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:20.640457 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:20.880866 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:20.880770 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:21.412658 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:21.412627 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:21.508601 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:21.508564 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:21.608814 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:21.608781 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:23.146206 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:23.146169 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:25.890793 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.890719 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:14:25.943401 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.943352 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-data\") pod \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " Jul 27 22:14:25.943401 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.943404 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istiod-ca-cert\") pod \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " Jul 27 22:14:25.943682 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.943449 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-envoy\") pod \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " Jul 27 22:14:25.943682 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.943474 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-podinfo\") pod \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " Jul 27 22:14:25.943682 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.943515 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-workload-socket\") pod \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " Jul 27 22:14:25.943682 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.943581 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5qnf\" (UniqueName: \"kubernetes.io/projected/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-kube-api-access-w5qnf\") pod \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " Jul 27 22:14:25.943682 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.943611 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-credential-socket\") pod \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " Jul 27 22:14:25.943682 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.943633 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-token\") pod \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " Jul 27 22:14:25.944031 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.943701 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-workload-certs\") pod \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\" (UID: \"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7\") " Jul 27 22:14:25.944031 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.943848 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" (UID: "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:14:25.944031 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.944010 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:14:25.944192 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.944086 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" (UID: "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:14:25.944192 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.944181 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" (UID: "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:14:25.944850 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.944135 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" (UID: "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:14:25.944850 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.944807 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" (UID: "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:14:25.946203 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.946155 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" (UID: "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:14:25.946412 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.946242 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" (UID: "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:14:25.946412 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.946251 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" (UID: "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:14:25.946557 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:25.946456 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-kube-api-access-w5qnf" (OuterVolumeSpecName: "kube-api-access-w5qnf") pod "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" (UID: "7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7"). InnerVolumeSpecName "kube-api-access-w5qnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:14:26.044764 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.044721 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5qnf\" (UniqueName: \"kubernetes.io/projected/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-kube-api-access-w5qnf\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:14:26.044764 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.044751 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:14:26.044764 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.044764 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:14:26.044764 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.044772 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:14:26.044764 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.044782 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:14:26.045181 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.044790 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:14:26.045181 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.044797 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:14:26.045181 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.044804 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:14:26.339435 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.339398 2561 generic.go:356] "Generic (PLEG): container finished" podID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" containerID="7d721813a44569f53097f8274b149682f481add7102e1c3aa9d2146e39cc013a" exitCode=0 Jul 27 22:14:26.339638 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.339449 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" Jul 27 22:14:26.339638 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.339479 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" event={"ID":"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7","Type":"ContainerDied","Data":"7d721813a44569f53097f8274b149682f481add7102e1c3aa9d2146e39cc013a"} Jul 27 22:14:26.339638 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.339534 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4" event={"ID":"7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7","Type":"ContainerDied","Data":"062352628a491be0a755d9448a271e919c580a0dedc6b42fe55e6fd0afd225f3"} Jul 27 22:14:26.339638 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.339558 2561 scope.go:117] "RemoveContainer" containerID="7d721813a44569f53097f8274b149682f481add7102e1c3aa9d2146e39cc013a" Jul 27 22:14:26.348740 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.348723 2561 scope.go:117] "RemoveContainer" containerID="b74f919c4476298400c85b11dd56e0a7f28b1eef34794b390e263d2700be3ffe" Jul 27 22:14:26.357019 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.357002 2561 scope.go:117] "RemoveContainer" containerID="7d721813a44569f53097f8274b149682f481add7102e1c3aa9d2146e39cc013a" Jul 27 22:14:26.357308 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:14:26.357283 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d721813a44569f53097f8274b149682f481add7102e1c3aa9d2146e39cc013a\": container with ID starting with 7d721813a44569f53097f8274b149682f481add7102e1c3aa9d2146e39cc013a not found: ID does not exist" containerID="7d721813a44569f53097f8274b149682f481add7102e1c3aa9d2146e39cc013a" Jul 27 22:14:26.357362 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.357319 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d721813a44569f53097f8274b149682f481add7102e1c3aa9d2146e39cc013a"} err="failed to get container status \"7d721813a44569f53097f8274b149682f481add7102e1c3aa9d2146e39cc013a\": rpc error: code = NotFound desc = could not find container \"7d721813a44569f53097f8274b149682f481add7102e1c3aa9d2146e39cc013a\": container with ID starting with 7d721813a44569f53097f8274b149682f481add7102e1c3aa9d2146e39cc013a not found: ID does not exist" Jul 27 22:14:26.357362 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.357337 2561 scope.go:117] "RemoveContainer" containerID="b74f919c4476298400c85b11dd56e0a7f28b1eef34794b390e263d2700be3ffe" Jul 27 22:14:26.357584 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:14:26.357565 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b74f919c4476298400c85b11dd56e0a7f28b1eef34794b390e263d2700be3ffe\": container with ID starting with b74f919c4476298400c85b11dd56e0a7f28b1eef34794b390e263d2700be3ffe not found: ID does not exist" containerID="b74f919c4476298400c85b11dd56e0a7f28b1eef34794b390e263d2700be3ffe" Jul 27 22:14:26.357639 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.357594 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74f919c4476298400c85b11dd56e0a7f28b1eef34794b390e263d2700be3ffe"} err="failed to get container status \"b74f919c4476298400c85b11dd56e0a7f28b1eef34794b390e263d2700be3ffe\": rpc error: code = NotFound desc = could not find container \"b74f919c4476298400c85b11dd56e0a7f28b1eef34794b390e263d2700be3ffe\": container with ID starting with b74f919c4476298400c85b11dd56e0a7f28b1eef34794b390e263d2700be3ffe not found: ID does not exist" Jul 27 22:14:26.360906 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.360883 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4"] Jul 27 22:14:26.365203 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.365183 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-78fe627e-openshift-default-6bf5c5c59b-rm9v4"] Jul 27 22:14:26.818614 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:26.818583 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" path="/var/lib/kubelet/pods/7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7/volumes" Jul 27 22:14:29.303996 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:29.303962 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:29.418583 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:29.418548 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:29.504039 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:29.504006 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:29.604416 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:29.604313 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:30.083012 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:30.082973 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:55.957764 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:14:55.957728 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:00.139482 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.139445 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753175-57cgb"] Jul 27 22:15:00.139867 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.139793 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" containerName="istio-proxy" Jul 27 22:15:00.139867 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.139809 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" containerName="istio-proxy" Jul 27 22:15:00.139867 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.139830 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0589f69d-01df-4378-a2a3-75df57c3d558" containerName="istio-proxy" Jul 27 22:15:00.139867 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.139835 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0589f69d-01df-4378-a2a3-75df57c3d558" containerName="istio-proxy" Jul 27 22:15:00.139867 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.139844 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0589f69d-01df-4378-a2a3-75df57c3d558" containerName="istio-proxy" Jul 27 22:15:00.139867 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.139850 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0589f69d-01df-4378-a2a3-75df57c3d558" containerName="istio-proxy" Jul 27 22:15:00.140061 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.139906 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" containerName="istio-proxy" Jul 27 22:15:00.140061 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.139915 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0589f69d-01df-4378-a2a3-75df57c3d558" containerName="istio-proxy" Jul 27 22:15:00.142835 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.142814 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-57cgb" Jul 27 22:15:00.145295 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.145273 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\"" Jul 27 22:15:00.152103 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.152079 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753175-57cgb"] Jul 27 22:15:00.154083 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.154066 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-57cgb" Jul 27 22:15:00.320088 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.319995 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753175-57cgb"] Jul 27 22:15:00.322167 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:15:00.322114 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e13204_85bb_47fd_bc7e_7cebb369ab1e.slice/crio-115a3860fc71dc8e867ff22bdaf8af1988195e90c12832ef7bece070ecac9c99 WatchSource:0}: Error finding container 115a3860fc71dc8e867ff22bdaf8af1988195e90c12832ef7bece070ecac9c99: Status 404 returned error can't find the container with id 115a3860fc71dc8e867ff22bdaf8af1988195e90c12832ef7bece070ecac9c99 Jul 27 22:15:00.324974 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.324936 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 22:15:00.513243 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:00.513208 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-57cgb" event={"ID":"b6e13204-85bb-47fd-bc7e-7cebb369ab1e","Type":"ContainerStarted","Data":"115a3860fc71dc8e867ff22bdaf8af1988195e90c12832ef7bece070ecac9c99"} Jul 27 22:15:02.594705 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:02.594653 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:02.706909 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:02.706876 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:02.808301 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:02.808204 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:03.532003 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:03.531963 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-57cgb" event={"ID":"b6e13204-85bb-47fd-bc7e-7cebb369ab1e","Type":"ContainerStarted","Data":"7a1f63c4f07f5c84d392e9b7e353cc784a5cdf955fb4b7c61dde14f83042e407"} Jul 27 22:15:03.551199 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:03.551144 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-57cgb" podStartSLOduration=1.082220859 podStartE2EDuration="3.55112991s" totalImagesPullingTime="2.468909051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:15:03.548987291 +0000 UTC m=+1781.166424610" watchObservedRunningTime="2026-07-27 22:15:03.55112991 +0000 UTC m=+1781.168567228" Jul 27 22:15:04.538775 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:04.538741 2561 generic.go:356] "Generic (PLEG): container finished" podID="b6e13204-85bb-47fd-bc7e-7cebb369ab1e" containerID="7a1f63c4f07f5c84d392e9b7e353cc784a5cdf955fb4b7c61dde14f83042e407" exitCode=0 Jul 27 22:15:04.539175 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:04.538790 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-57cgb" event={"ID":"b6e13204-85bb-47fd-bc7e-7cebb369ab1e","Type":"ContainerDied","Data":"7a1f63c4f07f5c84d392e9b7e353cc784a5cdf955fb4b7c61dde14f83042e407"} Jul 27 22:15:05.673945 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:05.673921 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-57cgb" Jul 27 22:15:06.549052 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:06.549026 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-57cgb" Jul 27 22:15:06.549234 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:06.549026 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-57cgb" event={"ID":"b6e13204-85bb-47fd-bc7e-7cebb369ab1e","Type":"ContainerDied","Data":"115a3860fc71dc8e867ff22bdaf8af1988195e90c12832ef7bece070ecac9c99"} Jul 27 22:15:06.549234 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:06.549142 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="115a3860fc71dc8e867ff22bdaf8af1988195e90c12832ef7bece070ecac9c99" Jul 27 22:15:18.717078 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:18.716966 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:24.506603 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:24.506566 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:24.620252 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:24.620210 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:24.715017 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:24.714976 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:35.244150 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:35.244115 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:36.085531 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:36.085489 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:42.496718 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:42.496677 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:42.615906 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:42.615867 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:42.714731 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:42.714686 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:42.808505 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:42.808404 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:43.666841 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:43.666800 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:43.765084 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:43.765047 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:43.858824 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:43.858767 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:45.086307 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:45.086255 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:45.186345 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:45.186308 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:45.607007 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:45.606972 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:52.168097 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:52.168056 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:52.273449 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:52.273408 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:52.906480 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:52.906445 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:58.894510 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:58.894477 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:58.962761 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:58.962711 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:59.149909 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.149873 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:59.434380 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.434300 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w"] Jul 27 22:15:59.434641 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.434628 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" containerName="istio-proxy" Jul 27 22:15:59.434685 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.434642 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" containerName="istio-proxy" Jul 27 22:15:59.434685 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.434657 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6e13204-85bb-47fd-bc7e-7cebb369ab1e" containerName="cleanup" Jul 27 22:15:59.434685 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.434663 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6e13204-85bb-47fd-bc7e-7cebb369ab1e" containerName="cleanup" Jul 27 22:15:59.434783 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.434718 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0589f69d-01df-4378-a2a3-75df57c3d558" containerName="istio-proxy" Jul 27 22:15:59.434783 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.434730 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d5b97a4-6898-4ba1-ae4e-e982c76cd9e7" containerName="istio-proxy" Jul 27 22:15:59.434783 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.434736 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6e13204-85bb-47fd-bc7e-7cebb369ab1e" containerName="cleanup" Jul 27 22:15:59.436645 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.436628 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.439486 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.439465 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2671d368-openshift-default-dockercfg-jj7wx\"" Jul 27 22:15:59.459191 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.459167 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w"] Jul 27 22:15:59.478413 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.478378 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:59.582726 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.582695 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-workload-socket\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.582909 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.582756 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istiod-ca-cert\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.582909 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.582804 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-podinfo\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.582909 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.582826 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-credential-socket\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.582909 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.582887 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-envoy\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.583069 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.582931 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc5ql\" (UniqueName: \"kubernetes.io/projected/711cecaf-3754-487b-9f6d-c3dce04f2a3d-kube-api-access-pc5ql\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.583069 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.582965 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-data\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.583069 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.582992 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-workload-certs\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.583069 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.583015 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-token\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.684321 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.684283 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istiod-ca-cert\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.684587 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.684357 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-podinfo\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.684587 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.684395 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-credential-socket\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.684587 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.684427 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-envoy\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.684587 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.684450 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc5ql\" (UniqueName: \"kubernetes.io/projected/711cecaf-3754-487b-9f6d-c3dce04f2a3d-kube-api-access-pc5ql\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.684587 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.684478 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-data\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.684587 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.684506 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-workload-certs\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.684587 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.684570 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-token\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.684975 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.684635 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-workload-socket\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.684975 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.684817 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-credential-socket\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.684975 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.684882 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-workload-certs\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.685149 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.685121 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-workload-socket\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.685297 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.685154 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-data\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.685698 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.685675 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istiod-ca-cert\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.686916 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.686889 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-envoy\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.687160 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.687142 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-podinfo\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.693794 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.693766 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-token\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.693933 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.693908 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc5ql\" (UniqueName: \"kubernetes.io/projected/711cecaf-3754-487b-9f6d-c3dce04f2a3d-kube-api-access-pc5ql\") pod \"e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.746654 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.746618 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:15:59.902096 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.902054 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w"] Jul 27 22:15:59.905362 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.905324 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:15:59.905483 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.905404 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:15:59.905483 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:15:59.905448 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:16:00.265126 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:00.265025 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:00.833415 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:00.833379 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" event={"ID":"711cecaf-3754-487b-9f6d-c3dce04f2a3d","Type":"ContainerStarted","Data":"2ae4dad2c066535c6d49fa876ccea91b2d69bb63bb08227b58fe6c5b50e858c2"} Jul 27 22:16:00.833641 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:00.833444 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" event={"ID":"711cecaf-3754-487b-9f6d-c3dce04f2a3d","Type":"ContainerStarted","Data":"b57551d9639fb9ad1be262e471e99277308ab223fb04f01855f5be883b76aacf"} Jul 27 22:16:00.853592 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:00.853538 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" podStartSLOduration=1.853520628 podStartE2EDuration="1.853520628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:16:00.851842143 +0000 UTC m=+1838.469279462" watchObservedRunningTime="2026-07-27 22:16:00.853520628 +0000 UTC m=+1838.470957950" Jul 27 22:16:01.747348 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:01.747311 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:16:01.753474 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:01.753425 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:16:01.838486 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:01.838454 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:16:01.839581 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:01.839557 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:16:02.907065 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:02.907031 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:03.013362 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:03.013320 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:03.182102 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:03.182001 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:03.944974 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:03.944929 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:04.110159 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:04.110128 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:04.216885 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:04.216792 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:07.326649 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:07.326613 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:08.010836 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:08.010800 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:08.105756 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:08.105715 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:08.457338 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:08.457298 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:14.386794 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:14.386754 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:14.488035 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:14.487994 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:16.840377 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:16.840332 2561 patch_prober.go:28] interesting pod/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 27 22:16:16.840858 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:16.840410 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 27 22:16:16.920133 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:16.920101 2561 generic.go:356] "Generic (PLEG): container finished" podID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerID="2ae4dad2c066535c6d49fa876ccea91b2d69bb63bb08227b58fe6c5b50e858c2" exitCode=0 Jul 27 22:16:16.920339 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:16.920163 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" event={"ID":"711cecaf-3754-487b-9f6d-c3dce04f2a3d","Type":"ContainerDied","Data":"2ae4dad2c066535c6d49fa876ccea91b2d69bb63bb08227b58fe6c5b50e858c2"} Jul 27 22:16:16.920651 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:16.920635 2561 scope.go:117] "RemoveContainer" containerID="2ae4dad2c066535c6d49fa876ccea91b2d69bb63bb08227b58fe6c5b50e858c2" Jul 27 22:16:16.921752 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:16.921723 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:16:16.921809 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:16.921790 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:16:16.921845 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:16.921826 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:16:17.930723 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:17.930678 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" event={"ID":"711cecaf-3754-487b-9f6d-c3dce04f2a3d","Type":"ContainerStarted","Data":"05d79a0506c556b546d30d6e6bec6c78ca5b1ec8e4ccb27f24e8c6cb31d8ca03"} Jul 27 22:16:18.747735 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:18.747685 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:16:18.747962 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:18.747871 2561 patch_prober.go:28] interesting pod/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 27 22:16:18.747962 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:18.747920 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 27 22:16:19.747210 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:19.747169 2561 patch_prober.go:28] interesting pod/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 27 22:16:19.747822 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:19.747788 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 27 22:16:20.747203 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:20.747155 2561 patch_prober.go:28] interesting pod/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 27 22:16:20.747444 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:20.747226 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 27 22:16:21.747777 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:21.747735 2561 patch_prober.go:28] interesting pod/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 27 22:16:21.748420 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:21.748383 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 27 22:16:22.751465 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:22.751433 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:16:22.751867 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:22.751754 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:16:22.752403 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:22.752387 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:16:23.680320 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:23.680278 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:24.860320 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:24.860283 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:24.943525 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:24.943494 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:25.024063 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:25.024018 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:29.888817 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:29.888782 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:30.299657 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:30.299616 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:36.162017 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:36.161980 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:36.263522 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:36.263485 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:36.443053 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:36.442960 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w"] Jul 27 22:16:36.443229 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:36.443184 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerName="istio-proxy" containerID="cri-o://05d79a0506c556b546d30d6e6bec6c78ca5b1ec8e4ccb27f24e8c6cb31d8ca03" gracePeriod=30 Jul 27 22:16:36.452311 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:36.452224 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:36.690766 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:36.690724 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:36.839111 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:36.839018 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:36.938464 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:36.938429 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:37.151473 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:37.151436 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:37.754040 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:37.753998 2561 patch_prober.go:28] interesting pod/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 22:16:37.754477 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:37.754086 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 22:16:41.696529 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.696494 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:16:41.869977 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.869878 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-workload-socket\") pod \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " Jul 27 22:16:41.869977 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.869917 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istiod-ca-cert\") pod \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " Jul 27 22:16:41.869977 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.869977 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-data\") pod \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " Jul 27 22:16:41.870297 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.869997 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-token\") pod \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " Jul 27 22:16:41.870297 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870018 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-credential-socket\") pod \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " Jul 27 22:16:41.870297 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870045 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-podinfo\") pod \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " Jul 27 22:16:41.870297 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870065 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc5ql\" (UniqueName: \"kubernetes.io/projected/711cecaf-3754-487b-9f6d-c3dce04f2a3d-kube-api-access-pc5ql\") pod \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " Jul 27 22:16:41.870297 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870098 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-envoy\") pod \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " Jul 27 22:16:41.870297 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870133 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-workload-certs\") pod \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\" (UID: \"711cecaf-3754-487b-9f6d-c3dce04f2a3d\") " Jul 27 22:16:41.870297 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870238 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "711cecaf-3754-487b-9f6d-c3dce04f2a3d" (UID: "711cecaf-3754-487b-9f6d-c3dce04f2a3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:16:41.870658 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870395 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "711cecaf-3754-487b-9f6d-c3dce04f2a3d" (UID: "711cecaf-3754-487b-9f6d-c3dce04f2a3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:16:41.870658 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870487 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "711cecaf-3754-487b-9f6d-c3dce04f2a3d" (UID: "711cecaf-3754-487b-9f6d-c3dce04f2a3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:16:41.870658 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870501 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "711cecaf-3754-487b-9f6d-c3dce04f2a3d" (UID: "711cecaf-3754-487b-9f6d-c3dce04f2a3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:16:41.870658 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870589 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:16:41.870658 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870604 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:16:41.870658 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870619 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:16:41.870658 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870630 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:16:41.870985 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.870884 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-data" (OuterVolumeSpecName: "istio-data") pod "711cecaf-3754-487b-9f6d-c3dce04f2a3d" (UID: "711cecaf-3754-487b-9f6d-c3dce04f2a3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:16:41.872455 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.872428 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "711cecaf-3754-487b-9f6d-c3dce04f2a3d" (UID: "711cecaf-3754-487b-9f6d-c3dce04f2a3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:16:41.872611 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.872486 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-token" (OuterVolumeSpecName: "istio-token") pod "711cecaf-3754-487b-9f6d-c3dce04f2a3d" (UID: "711cecaf-3754-487b-9f6d-c3dce04f2a3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:16:41.872800 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.872775 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "711cecaf-3754-487b-9f6d-c3dce04f2a3d" (UID: "711cecaf-3754-487b-9f6d-c3dce04f2a3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:16:41.872855 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.872821 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711cecaf-3754-487b-9f6d-c3dce04f2a3d-kube-api-access-pc5ql" (OuterVolumeSpecName: "kube-api-access-pc5ql") pod "711cecaf-3754-487b-9f6d-c3dce04f2a3d" (UID: "711cecaf-3754-487b-9f6d-c3dce04f2a3d"). InnerVolumeSpecName "kube-api-access-pc5ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:16:41.971404 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.971364 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:16:41.971404 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.971394 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc5ql\" (UniqueName: \"kubernetes.io/projected/711cecaf-3754-487b-9f6d-c3dce04f2a3d-kube-api-access-pc5ql\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:16:41.971404 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.971407 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:16:41.971404 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.971417 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:16:41.971676 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:41.971425 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711cecaf-3754-487b-9f6d-c3dce04f2a3d-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:16:42.066449 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.066409 2561 generic.go:356] "Generic (PLEG): container finished" podID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerID="05d79a0506c556b546d30d6e6bec6c78ca5b1ec8e4ccb27f24e8c6cb31d8ca03" exitCode=0 Jul 27 22:16:42.066627 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.066458 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" event={"ID":"711cecaf-3754-487b-9f6d-c3dce04f2a3d","Type":"ContainerDied","Data":"05d79a0506c556b546d30d6e6bec6c78ca5b1ec8e4ccb27f24e8c6cb31d8ca03"} Jul 27 22:16:42.066627 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.066467 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" Jul 27 22:16:42.066627 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.066515 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w" event={"ID":"711cecaf-3754-487b-9f6d-c3dce04f2a3d","Type":"ContainerDied","Data":"b57551d9639fb9ad1be262e471e99277308ab223fb04f01855f5be883b76aacf"} Jul 27 22:16:42.066627 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.066545 2561 scope.go:117] "RemoveContainer" containerID="05d79a0506c556b546d30d6e6bec6c78ca5b1ec8e4ccb27f24e8c6cb31d8ca03" Jul 27 22:16:42.075678 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.075659 2561 scope.go:117] "RemoveContainer" containerID="2ae4dad2c066535c6d49fa876ccea91b2d69bb63bb08227b58fe6c5b50e858c2" Jul 27 22:16:42.084340 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.084299 2561 scope.go:117] "RemoveContainer" containerID="05d79a0506c556b546d30d6e6bec6c78ca5b1ec8e4ccb27f24e8c6cb31d8ca03" Jul 27 22:16:42.084669 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:16:42.084647 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d79a0506c556b546d30d6e6bec6c78ca5b1ec8e4ccb27f24e8c6cb31d8ca03\": container with ID starting with 05d79a0506c556b546d30d6e6bec6c78ca5b1ec8e4ccb27f24e8c6cb31d8ca03 not found: ID does not exist" containerID="05d79a0506c556b546d30d6e6bec6c78ca5b1ec8e4ccb27f24e8c6cb31d8ca03" Jul 27 22:16:42.084750 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.084675 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d79a0506c556b546d30d6e6bec6c78ca5b1ec8e4ccb27f24e8c6cb31d8ca03"} err="failed to get container status \"05d79a0506c556b546d30d6e6bec6c78ca5b1ec8e4ccb27f24e8c6cb31d8ca03\": rpc error: code = NotFound desc = could not find container \"05d79a0506c556b546d30d6e6bec6c78ca5b1ec8e4ccb27f24e8c6cb31d8ca03\": container with ID starting with 05d79a0506c556b546d30d6e6bec6c78ca5b1ec8e4ccb27f24e8c6cb31d8ca03 not found: ID does not exist" Jul 27 22:16:42.084750 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.084692 2561 scope.go:117] "RemoveContainer" containerID="2ae4dad2c066535c6d49fa876ccea91b2d69bb63bb08227b58fe6c5b50e858c2" Jul 27 22:16:42.084923 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:16:42.084908 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae4dad2c066535c6d49fa876ccea91b2d69bb63bb08227b58fe6c5b50e858c2\": container with ID starting with 2ae4dad2c066535c6d49fa876ccea91b2d69bb63bb08227b58fe6c5b50e858c2 not found: ID does not exist" containerID="2ae4dad2c066535c6d49fa876ccea91b2d69bb63bb08227b58fe6c5b50e858c2" Jul 27 22:16:42.084960 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.084927 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae4dad2c066535c6d49fa876ccea91b2d69bb63bb08227b58fe6c5b50e858c2"} err="failed to get container status \"2ae4dad2c066535c6d49fa876ccea91b2d69bb63bb08227b58fe6c5b50e858c2\": rpc error: code = NotFound desc = could not find container \"2ae4dad2c066535c6d49fa876ccea91b2d69bb63bb08227b58fe6c5b50e858c2\": container with ID starting with 2ae4dad2c066535c6d49fa876ccea91b2d69bb63bb08227b58fe6c5b50e858c2 not found: ID does not exist" Jul 27 22:16:42.123053 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.122962 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w"] Jul 27 22:16:42.127770 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.127744 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2671d368-openshift-default-7c87c7d776-cz98w"] Jul 27 22:16:42.823865 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.823826 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" path="/var/lib/kubelet/pods/711cecaf-3754-487b-9f6d-c3dce04f2a3d/volumes" Jul 27 22:16:42.833206 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.833167 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:42.903430 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.903392 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:42.990907 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:42.990863 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:43.089494 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:43.089404 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:43.222205 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:43.222166 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:48.985848 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:48.985759 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:49.084889 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:49.084849 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:55.950891 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:55.950848 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:56.220749 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.220668 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj"] Jul 27 22:16:56.221044 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.221032 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerName="istio-proxy" Jul 27 22:16:56.221094 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.221047 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerName="istio-proxy" Jul 27 22:16:56.221126 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.221121 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerName="istio-proxy" Jul 27 22:16:56.221163 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.221131 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerName="istio-proxy" Jul 27 22:16:56.223106 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.223090 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.225425 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.225401 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5d6c9eb6-openshift-default-dockercfg-wgzld\"" Jul 27 22:16:56.238068 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.238040 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj"] Jul 27 22:16:56.253618 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.253579 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:56.299077 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.299034 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-envoy\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.299077 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.299081 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-workload-socket\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.299351 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.299101 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-credential-socket\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.299351 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.299162 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxs47\" (UniqueName: \"kubernetes.io/projected/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-kube-api-access-bxs47\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.299351 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.299234 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-podinfo\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.299351 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.299295 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-token\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.299351 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.299317 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-workload-certs\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.299351 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.299347 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istiod-ca-cert\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.299581 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.299379 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-data\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.400455 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.400418 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-envoy\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.400455 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.400462 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-workload-socket\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.400728 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.400480 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-credential-socket\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.400728 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.400618 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxs47\" (UniqueName: \"kubernetes.io/projected/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-kube-api-access-bxs47\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.400728 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.400688 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-podinfo\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.400728 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.400721 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-token\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.400932 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.400751 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-workload-certs\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.400932 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.400793 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istiod-ca-cert\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.400932 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.400845 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-data\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.401100 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.400935 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-credential-socket\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.401100 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.400962 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-workload-socket\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.401218 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.401161 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-data\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.401218 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.401190 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-workload-certs\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.401620 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.401599 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istiod-ca-cert\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.403004 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.402982 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-envoy\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.403110 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.403093 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-podinfo\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.414897 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.411584 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-token\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.414897 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.411890 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxs47\" (UniqueName: \"kubernetes.io/projected/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-kube-api-access-bxs47\") pod \"e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.526432 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.526328 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:56.536215 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.536176 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:56.676821 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.676795 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj"] Jul 27 22:16:56.679142 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:16:56.679112 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6030f1_f1e8_4b7a_bff2_5fed1d444e58.slice/crio-e4d6ee7d2a024db5560943d64552251d9e338c804537b1a26e2d5eb2ebce0b0e WatchSource:0}: Error finding container e4d6ee7d2a024db5560943d64552251d9e338c804537b1a26e2d5eb2ebce0b0e: Status 404 returned error can't find the container with id e4d6ee7d2a024db5560943d64552251d9e338c804537b1a26e2d5eb2ebce0b0e Jul 27 22:16:56.681310 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.681251 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:16:56.681394 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.681358 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:16:56.681433 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:56.681402 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:16:57.146905 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:57.146872 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" event={"ID":"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58","Type":"ContainerStarted","Data":"35ce8b914158fef2397a61709fe17a6112a0823706638572a788f52c04d0dd69"} Jul 27 22:16:57.147331 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:57.146924 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" event={"ID":"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58","Type":"ContainerStarted","Data":"e4d6ee7d2a024db5560943d64552251d9e338c804537b1a26e2d5eb2ebce0b0e"} Jul 27 22:16:57.174477 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:57.174423 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" podStartSLOduration=1.174407418 podStartE2EDuration="1.174407418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:16:57.172472404 +0000 UTC m=+1894.789909723" watchObservedRunningTime="2026-07-27 22:16:57.174407418 +0000 UTC m=+1894.791844737" Jul 27 22:16:57.536627 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:57.536578 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:57.541624 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:57.541596 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:58.152152 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:58.152120 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:58.153313 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:58.153295 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:16:59.135632 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:59.135594 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:59.259950 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:59.259913 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:59.414841 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:59.414804 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:16:59.511632 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:16:59.511597 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:17:00.128046 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:00.128007 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:17:00.217124 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:00.217082 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:17:03.653521 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:03.653472 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:17:13.309438 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:13.308952 2561 patch_prober.go:28] interesting pod/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:59710->10.133.0.51:15021: read: connection reset by peer" start-of-body= Jul 27 22:17:13.309438 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:13.309024 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:59710->10.133.0.51:15021: read: connection reset by peer" Jul 27 22:17:14.240218 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:14.240180 2561 generic.go:356] "Generic (PLEG): container finished" podID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerID="35ce8b914158fef2397a61709fe17a6112a0823706638572a788f52c04d0dd69" exitCode=0 Jul 27 22:17:14.240429 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:14.240254 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" event={"ID":"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58","Type":"ContainerDied","Data":"35ce8b914158fef2397a61709fe17a6112a0823706638572a788f52c04d0dd69"} Jul 27 22:17:14.240679 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:14.240666 2561 scope.go:117] "RemoveContainer" containerID="35ce8b914158fef2397a61709fe17a6112a0823706638572a788f52c04d0dd69" Jul 27 22:17:14.241754 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:14.241726 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:17:14.241834 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:14.241818 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:17:14.241879 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:14.241856 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:17:15.247708 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:15.247666 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" event={"ID":"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58","Type":"ContainerStarted","Data":"528eff077020fdee027c5cbde011964aae91ecdcdd55dd6cc003181c32d58a30"} Jul 27 22:17:15.537382 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:15.537166 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:17:15.537382 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:15.537221 2561 patch_prober.go:28] interesting pod/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 27 22:17:15.537382 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:15.537319 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 27 22:17:16.536952 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:16.536910 2561 patch_prober.go:28] interesting pod/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 27 22:17:16.537575 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:16.537542 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 27 22:17:17.537185 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:17.537144 2561 patch_prober.go:28] interesting pod/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 27 22:17:17.537830 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:17.537791 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 27 22:17:18.537592 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:18.537547 2561 patch_prober.go:28] interesting pod/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 27 22:17:18.538242 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:18.538205 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 27 22:17:19.537285 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:19.537214 2561 patch_prober.go:28] interesting pod/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 27 22:17:19.537506 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:19.537316 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 27 22:17:20.540369 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:20.540334 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:17:20.540797 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:20.540615 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:17:20.541370 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:20.541351 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:17:27.990337 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:17:27.990292 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:18:03.854025 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:03.853987 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf"] Jul 27 22:18:03.854684 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:03.854527 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerName="istio-proxy" Jul 27 22:18:03.854684 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:03.854549 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="711cecaf-3754-487b-9f6d-c3dce04f2a3d" containerName="istio-proxy" Jul 27 22:18:03.861704 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:03.861670 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf" Jul 27 22:18:03.865304 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:03.865242 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\"" Jul 27 22:18:03.865556 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:03.865305 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 22:18:03.865670 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:03.865325 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf"] Jul 27 22:18:03.912707 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:03.912663 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ebffc5-4855-4b28-9fce-824b85aeb96b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf\" (UID: \"43ebffc5-4855-4b28-9fce-824b85aeb96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf" Jul 27 22:18:04.013813 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:04.013773 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ebffc5-4855-4b28-9fce-824b85aeb96b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf\" (UID: \"43ebffc5-4855-4b28-9fce-824b85aeb96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf" Jul 27 22:18:04.014452 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:04.014430 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ebffc5-4855-4b28-9fce-824b85aeb96b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf\" (UID: \"43ebffc5-4855-4b28-9fce-824b85aeb96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf" Jul 27 22:18:04.175696 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:04.175649 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf" Jul 27 22:18:04.314107 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:04.314081 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf"] Jul 27 22:18:04.316914 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:18:04.316881 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ebffc5_4855_4b28_9fce_824b85aeb96b.slice/crio-54d6a31a635468fe65f0f40aba155026d59a2e94dfe1c07a24ee691c81c595b6 WatchSource:0}: Error finding container 54d6a31a635468fe65f0f40aba155026d59a2e94dfe1c07a24ee691c81c595b6: Status 404 returned error can't find the container with id 54d6a31a635468fe65f0f40aba155026d59a2e94dfe1c07a24ee691c81c595b6 Jul 27 22:18:04.493299 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:04.493204 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf" event={"ID":"43ebffc5-4855-4b28-9fce-824b85aeb96b","Type":"ContainerStarted","Data":"54d6a31a635468fe65f0f40aba155026d59a2e94dfe1c07a24ee691c81c595b6"} Jul 27 22:18:05.499972 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:05.499934 2561 generic.go:356] "Generic (PLEG): container finished" podID="43ebffc5-4855-4b28-9fce-824b85aeb96b" containerID="85dbc9a648e3ef4373c87137543bdc0fba05993eb7da5940b9b40d5538e8b9ed" exitCode=0 Jul 27 22:18:05.500495 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:05.500017 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf" event={"ID":"43ebffc5-4855-4b28-9fce-824b85aeb96b","Type":"ContainerDied","Data":"85dbc9a648e3ef4373c87137543bdc0fba05993eb7da5940b9b40d5538e8b9ed"} Jul 27 22:18:06.633922 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:06.633898 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf" Jul 27 22:18:06.737566 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:06.737525 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ebffc5-4855-4b28-9fce-824b85aeb96b-maas-api-service-ca\") pod \"43ebffc5-4855-4b28-9fce-824b85aeb96b\" (UID: \"43ebffc5-4855-4b28-9fce-824b85aeb96b\") " Jul 27 22:18:06.737892 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:06.737868 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ebffc5-4855-4b28-9fce-824b85aeb96b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43ebffc5-4855-4b28-9fce-824b85aeb96b" (UID: "43ebffc5-4855-4b28-9fce-824b85aeb96b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:18:06.839015 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:06.838984 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ebffc5-4855-4b28-9fce-824b85aeb96b-maas-api-service-ca\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:18:07.512120 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:07.512083 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf" event={"ID":"43ebffc5-4855-4b28-9fce-824b85aeb96b","Type":"ContainerDied","Data":"54d6a31a635468fe65f0f40aba155026d59a2e94dfe1c07a24ee691c81c595b6"} Jul 27 22:18:07.512314 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:07.512134 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54d6a31a635468fe65f0f40aba155026d59a2e94dfe1c07a24ee691c81c595b6" Jul 27 22:18:07.512314 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:07.512093 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf" Jul 27 22:18:19.310305 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:19.310175 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:18:19.413331 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:19.413286 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:18:19.513104 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:19.513069 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:18:23.856557 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:23.856524 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:18:23.939718 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:23.939682 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf"] Jul 27 22:18:23.942099 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:23.942062 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:18:23.942433 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:23.942414 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c9eb6-7d7mf"] Jul 27 22:18:24.818140 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:24.818104 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43ebffc5-4855-4b28-9fce-824b85aeb96b" path="/var/lib/kubelet/pods/43ebffc5-4855-4b28-9fce-824b85aeb96b/volumes" Jul 27 22:18:27.384999 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:27.384960 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:18:32.369359 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:18:32.369311 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:08.800804 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:08.800762 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:09.507590 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:09.507543 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:15.006247 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:15.006202 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:15.065073 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:15.065033 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:15.282659 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:15.282562 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj"] Jul 27 22:19:15.282833 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:15.282780 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerName="istio-proxy" containerID="cri-o://528eff077020fdee027c5cbde011964aae91ecdcdd55dd6cc003181c32d58a30" gracePeriod=30 Jul 27 22:19:15.305539 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:15.305504 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:15.541109 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:15.541010 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:16.632489 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:16.632448 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:16.731396 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:16.731355 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:16.830993 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:16.830959 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:17.624789 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:17.624752 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:17.726632 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:17.726595 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:17.827240 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:17.827206 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:19.254819 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:19.254772 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:20.536932 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.536905 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:19:20.600345 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.600232 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-podinfo\") pod \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " Jul 27 22:19:20.600345 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.600330 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-credential-socket\") pod \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " Jul 27 22:19:20.600564 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.600394 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxs47\" (UniqueName: \"kubernetes.io/projected/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-kube-api-access-bxs47\") pod \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " Jul 27 22:19:20.600564 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.600426 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-envoy\") pod \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " Jul 27 22:19:20.600564 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.600450 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-token\") pod \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " Jul 27 22:19:20.600564 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.600504 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-data\") pod \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " Jul 27 22:19:20.600564 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.600562 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istiod-ca-cert\") pod \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " Jul 27 22:19:20.600819 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.600597 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-workload-socket\") pod \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " Jul 27 22:19:20.600819 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.600621 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-workload-certs\") pod \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\" (UID: \"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58\") " Jul 27 22:19:20.600819 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.600656 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" (UID: "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:19:20.600988 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.600910 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:19:20.600988 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.600962 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" (UID: "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:19:20.601150 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.601115 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" (UID: "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:19:20.601226 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.601202 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" (UID: "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:19:20.601630 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.601607 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" (UID: "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:19:20.602565 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.602545 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" (UID: "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:19:20.602798 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.602781 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" (UID: "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:19:20.603027 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.603010 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" (UID: "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:19:20.603097 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.603027 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-kube-api-access-bxs47" (OuterVolumeSpecName: "kube-api-access-bxs47") pod "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" (UID: "0c6030f1-f1e8-4b7a-bff2-5fed1d444e58"). InnerVolumeSpecName "kube-api-access-bxs47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:19:20.701634 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.701591 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:19:20.701634 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.701626 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:19:20.701634 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.701636 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:19:20.701634 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.701644 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:19:20.701914 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.701653 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:19:20.701914 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.701661 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxs47\" (UniqueName: \"kubernetes.io/projected/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-kube-api-access-bxs47\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:19:20.701914 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.701672 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:19:20.701914 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.701681 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:19:20.892064 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.891978 2561 generic.go:356] "Generic (PLEG): container finished" podID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerID="528eff077020fdee027c5cbde011964aae91ecdcdd55dd6cc003181c32d58a30" exitCode=0 Jul 27 22:19:20.892064 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.892030 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" Jul 27 22:19:20.892064 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.892042 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" event={"ID":"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58","Type":"ContainerDied","Data":"528eff077020fdee027c5cbde011964aae91ecdcdd55dd6cc003181c32d58a30"} Jul 27 22:19:20.892325 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.892085 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj" event={"ID":"0c6030f1-f1e8-4b7a-bff2-5fed1d444e58","Type":"ContainerDied","Data":"e4d6ee7d2a024db5560943d64552251d9e338c804537b1a26e2d5eb2ebce0b0e"} Jul 27 22:19:20.892325 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.892106 2561 scope.go:117] "RemoveContainer" containerID="528eff077020fdee027c5cbde011964aae91ecdcdd55dd6cc003181c32d58a30" Jul 27 22:19:20.901229 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.901206 2561 scope.go:117] "RemoveContainer" containerID="35ce8b914158fef2397a61709fe17a6112a0823706638572a788f52c04d0dd69" Jul 27 22:19:20.909407 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.909381 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj"] Jul 27 22:19:20.909814 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.909794 2561 scope.go:117] "RemoveContainer" containerID="528eff077020fdee027c5cbde011964aae91ecdcdd55dd6cc003181c32d58a30" Jul 27 22:19:20.910086 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:19:20.910069 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528eff077020fdee027c5cbde011964aae91ecdcdd55dd6cc003181c32d58a30\": container with ID starting with 528eff077020fdee027c5cbde011964aae91ecdcdd55dd6cc003181c32d58a30 not found: ID does not exist" containerID="528eff077020fdee027c5cbde011964aae91ecdcdd55dd6cc003181c32d58a30" Jul 27 22:19:20.910156 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.910093 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528eff077020fdee027c5cbde011964aae91ecdcdd55dd6cc003181c32d58a30"} err="failed to get container status \"528eff077020fdee027c5cbde011964aae91ecdcdd55dd6cc003181c32d58a30\": rpc error: code = NotFound desc = could not find container \"528eff077020fdee027c5cbde011964aae91ecdcdd55dd6cc003181c32d58a30\": container with ID starting with 528eff077020fdee027c5cbde011964aae91ecdcdd55dd6cc003181c32d58a30 not found: ID does not exist" Jul 27 22:19:20.910156 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.910110 2561 scope.go:117] "RemoveContainer" containerID="35ce8b914158fef2397a61709fe17a6112a0823706638572a788f52c04d0dd69" Jul 27 22:19:20.910368 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:19:20.910349 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ce8b914158fef2397a61709fe17a6112a0823706638572a788f52c04d0dd69\": container with ID starting with 35ce8b914158fef2397a61709fe17a6112a0823706638572a788f52c04d0dd69 not found: ID does not exist" containerID="35ce8b914158fef2397a61709fe17a6112a0823706638572a788f52c04d0dd69" Jul 27 22:19:20.910413 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.910380 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ce8b914158fef2397a61709fe17a6112a0823706638572a788f52c04d0dd69"} err="failed to get container status \"35ce8b914158fef2397a61709fe17a6112a0823706638572a788f52c04d0dd69\": rpc error: code = NotFound desc = could not find container \"35ce8b914158fef2397a61709fe17a6112a0823706638572a788f52c04d0dd69\": container with ID starting with 35ce8b914158fef2397a61709fe17a6112a0823706638572a788f52c04d0dd69 not found: ID does not exist" Jul 27 22:19:20.913616 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:20.913593 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5d6c9eb6-openshift-default-6c877d58cb-dcfsj"] Jul 27 22:19:22.818375 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:22.818338 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" path="/var/lib/kubelet/pods/0c6030f1-f1e8-4b7a-bff2-5fed1d444e58/volumes" Jul 27 22:19:25.164388 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:25.164346 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:25.265652 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:25.265617 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:25.863892 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:25.863854 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:31.696299 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:31.696234 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:31.809021 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:31.808979 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:31.909172 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:31.909132 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:42.449795 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:42.449759 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:42.810662 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:42.810565 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:44.534851 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:44.534759 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:44.636355 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:19:44.636319 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:03.833005 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:03.832962 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:10.217710 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.217667 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:10.317504 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.317464 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:10.596460 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.596353 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:10.916333 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.916298 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn"] Jul 27 22:20:10.916742 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.916720 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerName="istio-proxy" Jul 27 22:20:10.916742 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.916740 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerName="istio-proxy" Jul 27 22:20:10.916926 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.916756 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ebffc5-4855-4b28-9fce-824b85aeb96b" containerName="revoke-keys" Jul 27 22:20:10.916926 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.916762 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ebffc5-4855-4b28-9fce-824b85aeb96b" containerName="revoke-keys" Jul 27 22:20:10.916926 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.916835 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerName="istio-proxy" Jul 27 22:20:10.916926 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.916844 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerName="istio-proxy" Jul 27 22:20:10.916926 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.916855 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ebffc5-4855-4b28-9fce-824b85aeb96b" containerName="revoke-keys" Jul 27 22:20:10.919583 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.919560 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:10.919719 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.919676 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:10.921820 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.921801 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7a6dea-openshift-default-dockercfg-qhrv2\"" Jul 27 22:20:10.932059 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.932029 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn"] Jul 27 22:20:10.946242 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.946203 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-credential-socket\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:10.946437 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.946283 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-data\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:10.946437 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.946332 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4rqd\" (UniqueName: \"kubernetes.io/projected/1069ccba-9c2a-434d-b832-6d460e231ffc-kube-api-access-q4rqd\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:10.946437 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.946389 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-podinfo\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:10.946437 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.946416 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1069ccba-9c2a-434d-b832-6d460e231ffc-istiod-ca-cert\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:10.946596 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.946455 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-envoy\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:10.946596 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.946507 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-token\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:10.946596 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.946549 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-workload-certs\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:10.946705 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:10.946592 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-workload-socket\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.047794 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.047748 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-podinfo\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.047794 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.047794 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1069ccba-9c2a-434d-b832-6d460e231ffc-istiod-ca-cert\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.048054 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.047830 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-envoy\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.048054 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.047857 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-token\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.048054 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.047886 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-workload-certs\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.048054 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.047921 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-workload-socket\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.048054 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.047954 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-credential-socket\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.048054 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.048000 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-data\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.048054 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.048040 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4rqd\" (UniqueName: \"kubernetes.io/projected/1069ccba-9c2a-434d-b832-6d460e231ffc-kube-api-access-q4rqd\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.048479 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.048453 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-workload-certs\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.048627 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.048611 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-workload-socket\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.048690 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.048623 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-data\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.048749 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.048697 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-credential-socket\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.048889 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.048864 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1069ccba-9c2a-434d-b832-6d460e231ffc-istiod-ca-cert\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.050252 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.050232 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-envoy\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.050520 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.050502 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-podinfo\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.054327 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.054251 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:11.055757 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.055732 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4rqd\" (UniqueName: \"kubernetes.io/projected/1069ccba-9c2a-434d-b832-6d460e231ffc-kube-api-access-q4rqd\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.055843 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.055805 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-token\") pod \"e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.231640 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.231538 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:11.375579 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.375546 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn"] Jul 27 22:20:11.377904 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:20:11.377876 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1069ccba_9c2a_434d_b832_6d460e231ffc.slice/crio-745db7f73622389320df0ddcd28aed5db6b2cc2316e184716b369b1c494b7063 WatchSource:0}: Error finding container 745db7f73622389320df0ddcd28aed5db6b2cc2316e184716b369b1c494b7063: Status 404 returned error can't find the container with id 745db7f73622389320df0ddcd28aed5db6b2cc2316e184716b369b1c494b7063 Jul 27 22:20:11.380014 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.379982 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:20:11.380073 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.380055 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:20:11.380113 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:11.380084 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:20:12.174896 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:12.174859 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" event={"ID":"1069ccba-9c2a-434d-b832-6d460e231ffc","Type":"ContainerStarted","Data":"23707cefabbb73f3b36434eb5941cb117108c9c765e92596bd1f607c78713482"} Jul 27 22:20:12.175090 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:12.174913 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" event={"ID":"1069ccba-9c2a-434d-b832-6d460e231ffc","Type":"ContainerStarted","Data":"745db7f73622389320df0ddcd28aed5db6b2cc2316e184716b369b1c494b7063"} Jul 27 22:20:12.199934 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:12.199880 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" podStartSLOduration=2.199861847 podStartE2EDuration="2.199861847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:20:12.197696183 +0000 UTC m=+2089.815133501" watchObservedRunningTime="2026-07-27 22:20:12.199861847 +0000 UTC m=+2089.817299165" Jul 27 22:20:12.231852 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:12.231806 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:12.236827 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:12.236801 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:13.180497 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:13.180468 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:13.181441 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:13.181421 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:14.177200 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:14.177153 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:14.277585 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:14.277543 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:14.372016 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:14.371975 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:14.662747 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:14.662704 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:14.859747 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:14.859712 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:14.957588 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:14.957488 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:18.696785 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:18.696751 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:19.058191 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.058097 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:19.362204 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.362106 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7"] Jul 27 22:20:19.362639 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.362624 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerName="istio-proxy" Jul 27 22:20:19.362952 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.362643 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6030f1-f1e8-4b7a-bff2-5fed1d444e58" containerName="istio-proxy" Jul 27 22:20:19.366294 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.366274 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.368641 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.368621 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-00544d-openshift-default-dockercfg-wf9h9\"" Jul 27 22:20:19.379791 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.379763 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7"] Jul 27 22:20:19.392287 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.392229 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:19.424562 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.424520 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istiod-ca-cert\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.424783 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.424600 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-envoy\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.424783 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.424628 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-data\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.424783 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.424659 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-podinfo\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.424783 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.424683 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrcjq\" (UniqueName: \"kubernetes.io/projected/c66dd5b6-7e52-4d25-a3bb-4824049354b4-kube-api-access-qrcjq\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.424783 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.424743 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-credential-socket\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.425014 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.424799 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-workload-socket\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.425014 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.424864 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-workload-certs\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.425014 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.424889 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-token\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.501299 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.501233 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:19.525651 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.525611 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-envoy\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.525651 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.525653 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-data\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.525912 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.525702 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-podinfo\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.525912 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.525727 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrcjq\" (UniqueName: \"kubernetes.io/projected/c66dd5b6-7e52-4d25-a3bb-4824049354b4-kube-api-access-qrcjq\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.525912 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.525753 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-credential-socket\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.525912 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.525786 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-workload-socket\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.525912 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.525848 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-workload-certs\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.525912 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.525870 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-token\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.526215 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.525950 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istiod-ca-cert\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.526215 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.526150 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-data\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.526384 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.526232 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-credential-socket\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.526496 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.526474 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-workload-certs\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.526586 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.526561 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-workload-socket\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.526794 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.526775 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istiod-ca-cert\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.528033 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.528012 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-envoy\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.528192 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.528173 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-podinfo\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.534216 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.534194 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-token\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.534363 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.534342 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrcjq\" (UniqueName: \"kubernetes.io/projected/c66dd5b6-7e52-4d25-a3bb-4824049354b4-kube-api-access-qrcjq\") pod \"e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.676985 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.676948 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:19.816883 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.816849 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7"] Jul 27 22:20:19.818953 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:20:19.818923 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66dd5b6_7e52_4d25_a3bb_4824049354b4.slice/crio-e9b05605548ecd727c6fb63b3253165a668c566677b3e765b343dd177833d00d WatchSource:0}: Error finding container e9b05605548ecd727c6fb63b3253165a668c566677b3e765b343dd177833d00d: Status 404 returned error can't find the container with id e9b05605548ecd727c6fb63b3253165a668c566677b3e765b343dd177833d00d Jul 27 22:20:19.821627 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.821587 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:20:19.821716 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.821679 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:20:19.821752 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:19.821727 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:20:20.217695 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:20.217655 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" event={"ID":"c66dd5b6-7e52-4d25-a3bb-4824049354b4","Type":"ContainerStarted","Data":"e5922dddd7e8ffbf1108f6871bbb6bacdd868b271cfb2b82e6a3020cba9122c2"} Jul 27 22:20:20.217883 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:20.217719 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" event={"ID":"c66dd5b6-7e52-4d25-a3bb-4824049354b4","Type":"ContainerStarted","Data":"e9b05605548ecd727c6fb63b3253165a668c566677b3e765b343dd177833d00d"} Jul 27 22:20:20.240286 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:20.240204 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" podStartSLOduration=1.240179145 podStartE2EDuration="1.240179145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:20:20.23808594 +0000 UTC m=+2097.855523260" watchObservedRunningTime="2026-07-27 22:20:20.240179145 +0000 UTC m=+2097.857616465" Jul 27 22:20:20.677143 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:20.677102 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:20.682382 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:20.682352 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:21.223069 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:21.223037 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:21.224229 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:21.224210 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:22.186564 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:22.186527 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:22.284955 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:22.284922 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:22.489116 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:22.489029 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:22.859041 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:22.858942 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:22.957433 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:22.957396 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:23.057786 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:23.057752 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:26.697173 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:26.697133 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:27.283062 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:27.283027 2561 generic.go:356] "Generic (PLEG): container finished" podID="1069ccba-9c2a-434d-b832-6d460e231ffc" containerID="23707cefabbb73f3b36434eb5941cb117108c9c765e92596bd1f607c78713482" exitCode=0 Jul 27 22:20:27.283358 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:27.283108 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" event={"ID":"1069ccba-9c2a-434d-b832-6d460e231ffc","Type":"ContainerDied","Data":"23707cefabbb73f3b36434eb5941cb117108c9c765e92596bd1f607c78713482"} Jul 27 22:20:27.283601 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:27.283585 2561 scope.go:117] "RemoveContainer" containerID="23707cefabbb73f3b36434eb5941cb117108c9c765e92596bd1f607c78713482" Jul 27 22:20:27.284691 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:27.284667 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:20:27.284756 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:27.284742 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:20:27.284796 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:27.284782 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:20:28.181237 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:28.181194 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:28.290674 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:28.290637 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" event={"ID":"1069ccba-9c2a-434d-b832-6d460e231ffc","Type":"ContainerStarted","Data":"db8eed7fd4da5df8d49ad4c9af23c20be1db4570c0fbd68247d461eeb3ad865e"} Jul 27 22:20:29.010827 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:29.010783 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:29.232159 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:29.232123 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:29.233405 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:29.233368 2561 patch_prober.go:28] interesting pod/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 27 22:20:29.233528 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:29.233433 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" podUID="1069ccba-9c2a-434d-b832-6d460e231ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 27 22:20:30.233141 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:30.232685 2561 patch_prober.go:28] interesting pod/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 27 22:20:30.233141 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:30.232750 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" podUID="1069ccba-9c2a-434d-b832-6d460e231ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 27 22:20:31.233584 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:31.233061 2561 patch_prober.go:28] interesting pod/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 27 22:20:31.233584 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:31.233148 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" podUID="1069ccba-9c2a-434d-b832-6d460e231ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 27 22:20:32.232755 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:32.232714 2561 patch_prober.go:28] interesting pod/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 27 22:20:32.233070 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:32.233040 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" podUID="1069ccba-9c2a-434d-b832-6d460e231ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 27 22:20:33.088414 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:33.088385 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:33.238322 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:33.238280 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:33.239159 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:33.239126 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:33.240700 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:33.240679 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:20:37.264514 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:37.264462 2561 patch_prober.go:28] interesting pod/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 22:20:37.264892 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:37.264552 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 22:20:38.354187 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:38.354145 2561 generic.go:356] "Generic (PLEG): container finished" podID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerID="e5922dddd7e8ffbf1108f6871bbb6bacdd868b271cfb2b82e6a3020cba9122c2" exitCode=0 Jul 27 22:20:38.354837 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:38.354288 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" event={"ID":"c66dd5b6-7e52-4d25-a3bb-4824049354b4","Type":"ContainerDied","Data":"e5922dddd7e8ffbf1108f6871bbb6bacdd868b271cfb2b82e6a3020cba9122c2"} Jul 27 22:20:38.355447 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:38.355424 2561 scope.go:117] "RemoveContainer" containerID="e5922dddd7e8ffbf1108f6871bbb6bacdd868b271cfb2b82e6a3020cba9122c2" Jul 27 22:20:38.356988 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:38.356955 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:20:38.357072 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:38.357029 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:20:38.357124 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:38.357077 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:20:39.361855 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:39.361816 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" event={"ID":"c66dd5b6-7e52-4d25-a3bb-4824049354b4","Type":"ContainerStarted","Data":"c315d2ebd37b876d5e154268461546a292acd9b925bc2815772be4f056965f42"} Jul 27 22:20:39.678073 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:39.678031 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:39.678326 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:39.678300 2561 patch_prober.go:28] interesting pod/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 27 22:20:39.678380 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:39.678352 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 27 22:20:40.677781 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:40.677742 2561 patch_prober.go:28] interesting pod/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 27 22:20:40.678194 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:40.677804 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 27 22:20:41.677358 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:41.677314 2561 patch_prober.go:28] interesting pod/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 27 22:20:41.677589 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:41.677377 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 27 22:20:42.677698 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:42.677661 2561 patch_prober.go:28] interesting pod/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 27 22:20:42.678111 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:42.677718 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 27 22:20:43.681047 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:43.681012 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:43.681481 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:43.681373 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:20:43.682019 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:20:43.682003 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:21:17.028012 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:21:17.027915 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:56.908400 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:21:56.908349 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:57.850368 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:21:57.850333 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:08.610586 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:08.610540 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:08.706837 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:08.706800 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:13.139109 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:13.139070 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:13.628575 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:13.628543 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:19.535983 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:19.535941 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:19.606789 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:19.606752 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:19.792069 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:19.791977 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn"] Jul 27 22:22:19.792384 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:19.792249 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" podUID="1069ccba-9c2a-434d-b832-6d460e231ffc" containerName="istio-proxy" containerID="cri-o://db8eed7fd4da5df8d49ad4c9af23c20be1db4570c0fbd68247d461eeb3ad865e" gracePeriod=30 Jul 27 22:22:19.827384 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:19.827343 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:20.058715 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:20.058621 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:24.931363 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:24.931329 2561 generic.go:356] "Generic (PLEG): container finished" podID="1069ccba-9c2a-434d-b832-6d460e231ffc" containerID="db8eed7fd4da5df8d49ad4c9af23c20be1db4570c0fbd68247d461eeb3ad865e" exitCode=0 Jul 27 22:22:24.931805 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:24.931384 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" event={"ID":"1069ccba-9c2a-434d-b832-6d460e231ffc","Type":"ContainerDied","Data":"db8eed7fd4da5df8d49ad4c9af23c20be1db4570c0fbd68247d461eeb3ad865e"} Jul 27 22:22:24.931805 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:24.931439 2561 scope.go:117] "RemoveContainer" containerID="23707cefabbb73f3b36434eb5941cb117108c9c765e92596bd1f607c78713482" Jul 27 22:22:25.056002 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.055934 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:22:25.150852 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.150817 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4rqd\" (UniqueName: \"kubernetes.io/projected/1069ccba-9c2a-434d-b832-6d460e231ffc-kube-api-access-q4rqd\") pod \"1069ccba-9c2a-434d-b832-6d460e231ffc\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " Jul 27 22:22:25.150852 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.150859 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1069ccba-9c2a-434d-b832-6d460e231ffc-istiod-ca-cert\") pod \"1069ccba-9c2a-434d-b832-6d460e231ffc\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " Jul 27 22:22:25.151133 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.150909 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-envoy\") pod \"1069ccba-9c2a-434d-b832-6d460e231ffc\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " Jul 27 22:22:25.151133 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.151064 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-data\") pod \"1069ccba-9c2a-434d-b832-6d460e231ffc\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " Jul 27 22:22:25.151133 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.151102 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-token\") pod \"1069ccba-9c2a-434d-b832-6d460e231ffc\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " Jul 27 22:22:25.151133 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.151125 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-workload-socket\") pod \"1069ccba-9c2a-434d-b832-6d460e231ffc\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " Jul 27 22:22:25.151398 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.151181 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-podinfo\") pod \"1069ccba-9c2a-434d-b832-6d460e231ffc\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " Jul 27 22:22:25.151398 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.151232 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-workload-certs\") pod \"1069ccba-9c2a-434d-b832-6d460e231ffc\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " Jul 27 22:22:25.151398 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.151307 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-credential-socket\") pod \"1069ccba-9c2a-434d-b832-6d460e231ffc\" (UID: \"1069ccba-9c2a-434d-b832-6d460e231ffc\") " Jul 27 22:22:25.151622 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.151487 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1069ccba-9c2a-434d-b832-6d460e231ffc" (UID: "1069ccba-9c2a-434d-b832-6d460e231ffc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:25.151622 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.151593 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1069ccba-9c2a-434d-b832-6d460e231ffc" (UID: "1069ccba-9c2a-434d-b832-6d460e231ffc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:25.151730 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.151658 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:25.151730 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.151708 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1069ccba-9c2a-434d-b832-6d460e231ffc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1069ccba-9c2a-434d-b832-6d460e231ffc" (UID: "1069ccba-9c2a-434d-b832-6d460e231ffc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:22:25.151816 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.151727 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1069ccba-9c2a-434d-b832-6d460e231ffc" (UID: "1069ccba-9c2a-434d-b832-6d460e231ffc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:25.151967 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.151896 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-data" (OuterVolumeSpecName: "istio-data") pod "1069ccba-9c2a-434d-b832-6d460e231ffc" (UID: "1069ccba-9c2a-434d-b832-6d460e231ffc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:25.153172 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.153147 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1069ccba-9c2a-434d-b832-6d460e231ffc-kube-api-access-q4rqd" (OuterVolumeSpecName: "kube-api-access-q4rqd") pod "1069ccba-9c2a-434d-b832-6d460e231ffc" (UID: "1069ccba-9c2a-434d-b832-6d460e231ffc"). InnerVolumeSpecName "kube-api-access-q4rqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:22:25.153480 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.153456 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1069ccba-9c2a-434d-b832-6d460e231ffc" (UID: "1069ccba-9c2a-434d-b832-6d460e231ffc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:22:25.153645 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.153628 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1069ccba-9c2a-434d-b832-6d460e231ffc" (UID: "1069ccba-9c2a-434d-b832-6d460e231ffc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:25.153713 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.153660 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-token" (OuterVolumeSpecName: "istio-token") pod "1069ccba-9c2a-434d-b832-6d460e231ffc" (UID: "1069ccba-9c2a-434d-b832-6d460e231ffc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:22:25.252381 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.252342 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:25.252381 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.252373 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:25.252381 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.252385 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:25.252655 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.252397 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4rqd\" (UniqueName: \"kubernetes.io/projected/1069ccba-9c2a-434d-b832-6d460e231ffc-kube-api-access-q4rqd\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:25.252655 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.252410 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1069ccba-9c2a-434d-b832-6d460e231ffc-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:25.252655 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.252421 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:25.252655 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.252431 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:25.252655 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.252442 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1069ccba-9c2a-434d-b832-6d460e231ffc-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:25.937571 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.937539 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" Jul 27 22:22:25.937571 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.937559 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn" event={"ID":"1069ccba-9c2a-434d-b832-6d460e231ffc","Type":"ContainerDied","Data":"745db7f73622389320df0ddcd28aed5db6b2cc2316e184716b369b1c494b7063"} Jul 27 22:22:25.938049 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.937621 2561 scope.go:117] "RemoveContainer" containerID="db8eed7fd4da5df8d49ad4c9af23c20be1db4570c0fbd68247d461eeb3ad865e" Jul 27 22:22:25.963061 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.963031 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn"] Jul 27 22:22:25.965897 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:25.965873 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7a6dea-openshift-default-6647f66554-l9hgn"] Jul 27 22:22:26.818243 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:26.818208 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1069ccba-9c2a-434d-b832-6d460e231ffc" path="/var/lib/kubelet/pods/1069ccba-9c2a-434d-b832-6d460e231ffc/volumes" Jul 27 22:22:30.708684 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:30.708643 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:31.969436 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:31.969395 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:35.324589 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:35.324548 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:35.750420 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:35.750382 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:41.796590 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:41.796552 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:41.927859 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:41.927826 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:42.078634 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:42.078554 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7"] Jul 27 22:22:42.078783 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:42.078750 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerName="istio-proxy" containerID="cri-o://c315d2ebd37b876d5e154268461546a292acd9b925bc2815772be4f056965f42" gracePeriod=30 Jul 27 22:22:42.107103 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:42.107066 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:42.373354 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:42.373243 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:42.769347 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:42.769294 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:43.178185 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.178148 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq"] Jul 27 22:22:43.178692 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.178565 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1069ccba-9c2a-434d-b832-6d460e231ffc" containerName="istio-proxy" Jul 27 22:22:43.178692 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.178579 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1069ccba-9c2a-434d-b832-6d460e231ffc" containerName="istio-proxy" Jul 27 22:22:43.178692 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.178644 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1069ccba-9c2a-434d-b832-6d460e231ffc" containerName="istio-proxy" Jul 27 22:22:43.180883 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.180864 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.214803 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.214776 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-721ee8-openshift-default-dockercfg-4w4wk\"" Jul 27 22:22:43.222844 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.222813 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq"] Jul 27 22:22:43.227963 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.227937 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:43.316836 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.316787 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-credential-socket\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.317038 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.316849 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf843206-c265-41bd-928d-c8e73d328ac8-istio-podinfo\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.317038 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.316885 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf843206-c265-41bd-928d-c8e73d328ac8-istiod-ca-cert\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.317038 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.316911 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-workload-certs\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.317038 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.316944 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-istio-envoy\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.317038 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.316968 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqht\" (UniqueName: \"kubernetes.io/projected/cf843206-c265-41bd-928d-c8e73d328ac8-kube-api-access-pnqht\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.317038 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.316994 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-istio-data\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.317366 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.317078 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf843206-c265-41bd-928d-c8e73d328ac8-istio-token\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.317366 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.317180 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-workload-socket\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.417780 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.417744 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-workload-socket\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.417984 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.417822 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-credential-socket\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.417984 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.417864 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf843206-c265-41bd-928d-c8e73d328ac8-istio-podinfo\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.417984 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.417898 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf843206-c265-41bd-928d-c8e73d328ac8-istiod-ca-cert\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.417984 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.417923 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-workload-certs\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.417984 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.417956 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-istio-envoy\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.417984 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.417978 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqht\" (UniqueName: \"kubernetes.io/projected/cf843206-c265-41bd-928d-c8e73d328ac8-kube-api-access-pnqht\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.418328 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.418012 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-istio-data\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.418328 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.418053 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf843206-c265-41bd-928d-c8e73d328ac8-istio-token\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.418328 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.418192 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-workload-socket\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.418614 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.418549 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-credential-socket\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.419029 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.418970 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-workload-certs\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.419163 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.419128 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf843206-c265-41bd-928d-c8e73d328ac8-istiod-ca-cert\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.419237 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.419177 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-istio-data\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.421452 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.421406 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-istio-envoy\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.421590 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.421473 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf843206-c265-41bd-928d-c8e73d328ac8-istio-podinfo\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.437797 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.437724 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf843206-c265-41bd-928d-c8e73d328ac8-istio-token\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.450184 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.450149 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqht\" (UniqueName: \"kubernetes.io/projected/cf843206-c265-41bd-928d-c8e73d328ac8-kube-api-access-pnqht\") pod \"e2e-api-a-721ee8-openshift-default-56db875588-r8gdq\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.490925 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.490882 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:43.623090 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.623039 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:43.636844 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.636808 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq"] Jul 27 22:22:43.640647 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:22:43.640623 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf843206_c265_41bd_928d_c8e73d328ac8.slice/crio-bf5ec698c197aa44221ce0405055e96c1140d4895ae0cf06557fd3fd55b9d565 WatchSource:0}: Error finding container bf5ec698c197aa44221ce0405055e96c1140d4895ae0cf06557fd3fd55b9d565: Status 404 returned error can't find the container with id bf5ec698c197aa44221ce0405055e96c1140d4895ae0cf06557fd3fd55b9d565 Jul 27 22:22:43.642612 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.642582 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:22:43.642695 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.642644 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:22:43.642695 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.642677 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:22:43.683753 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.683714 2561 patch_prober.go:28] interesting pod/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 22:22:43.683908 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:43.683781 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 22:22:44.033579 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:44.033480 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" event={"ID":"cf843206-c265-41bd-928d-c8e73d328ac8","Type":"ContainerStarted","Data":"97c52d9a91eac04dee3944a47075f2106d006e450de88047de39ac77e44ca8ff"} Jul 27 22:22:44.033579 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:44.033538 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" event={"ID":"cf843206-c265-41bd-928d-c8e73d328ac8","Type":"ContainerStarted","Data":"bf5ec698c197aa44221ce0405055e96c1140d4895ae0cf06557fd3fd55b9d565"} Jul 27 22:22:44.055924 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:44.055871 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" podStartSLOduration=1.055851245 podStartE2EDuration="1.055851245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:22:44.053446716 +0000 UTC m=+2241.670884036" watchObservedRunningTime="2026-07-27 22:22:44.055851245 +0000 UTC m=+2241.673288563" Jul 27 22:22:44.491502 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:44.491425 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:44.496239 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:44.496213 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:45.039424 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:45.039390 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:45.040461 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:45.040440 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:22:46.134976 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:46.134943 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:46.235479 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:46.235445 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:46.353835 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:46.353803 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:46.480931 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:46.480896 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:47.039864 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.039820 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:47.137107 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.137078 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:47.339020 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.338992 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:22:47.455955 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.455919 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-data\") pod \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " Jul 27 22:22:47.455955 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.455960 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-podinfo\") pod \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " Jul 27 22:22:47.456190 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.455992 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-credential-socket\") pod \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " Jul 27 22:22:47.456190 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456023 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istiod-ca-cert\") pod \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " Jul 27 22:22:47.456190 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456048 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-workload-socket\") pod \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " Jul 27 22:22:47.456190 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456095 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-workload-certs\") pod \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " Jul 27 22:22:47.456190 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456180 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-envoy\") pod \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " Jul 27 22:22:47.456493 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456210 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrcjq\" (UniqueName: \"kubernetes.io/projected/c66dd5b6-7e52-4d25-a3bb-4824049354b4-kube-api-access-qrcjq\") pod \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " Jul 27 22:22:47.456493 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456233 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-token\") pod \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\" (UID: \"c66dd5b6-7e52-4d25-a3bb-4824049354b4\") " Jul 27 22:22:47.456493 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456433 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c66dd5b6-7e52-4d25-a3bb-4824049354b4" (UID: "c66dd5b6-7e52-4d25-a3bb-4824049354b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:47.456493 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456445 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c66dd5b6-7e52-4d25-a3bb-4824049354b4" (UID: "c66dd5b6-7e52-4d25-a3bb-4824049354b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:47.456493 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456457 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c66dd5b6-7e52-4d25-a3bb-4824049354b4" (UID: "c66dd5b6-7e52-4d25-a3bb-4824049354b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:47.456493 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456480 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c66dd5b6-7e52-4d25-a3bb-4824049354b4" (UID: "c66dd5b6-7e52-4d25-a3bb-4824049354b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:22:47.456793 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456657 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:47.456793 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456677 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:47.456793 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456695 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:47.456793 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.456708 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:47.457127 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.457108 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-data" (OuterVolumeSpecName: "istio-data") pod "c66dd5b6-7e52-4d25-a3bb-4824049354b4" (UID: "c66dd5b6-7e52-4d25-a3bb-4824049354b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:47.458828 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.458804 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c66dd5b6-7e52-4d25-a3bb-4824049354b4" (UID: "c66dd5b6-7e52-4d25-a3bb-4824049354b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:22:47.458907 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.458804 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c66dd5b6-7e52-4d25-a3bb-4824049354b4" (UID: "c66dd5b6-7e52-4d25-a3bb-4824049354b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:47.458976 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.458911 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-token" (OuterVolumeSpecName: "istio-token") pod "c66dd5b6-7e52-4d25-a3bb-4824049354b4" (UID: "c66dd5b6-7e52-4d25-a3bb-4824049354b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:22:47.458976 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.458940 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66dd5b6-7e52-4d25-a3bb-4824049354b4-kube-api-access-qrcjq" (OuterVolumeSpecName: "kube-api-access-qrcjq") pod "c66dd5b6-7e52-4d25-a3bb-4824049354b4" (UID: "c66dd5b6-7e52-4d25-a3bb-4824049354b4"). InnerVolumeSpecName "kube-api-access-qrcjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:22:47.558046 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.558010 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:47.558046 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.558043 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:47.558244 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.558076 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrcjq\" (UniqueName: \"kubernetes.io/projected/c66dd5b6-7e52-4d25-a3bb-4824049354b4-kube-api-access-qrcjq\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:47.558244 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.558086 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:47.558244 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:47.558095 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c66dd5b6-7e52-4d25-a3bb-4824049354b4-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:22:48.055688 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:48.055651 2561 generic.go:356] "Generic (PLEG): container finished" podID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerID="c315d2ebd37b876d5e154268461546a292acd9b925bc2815772be4f056965f42" exitCode=0 Jul 27 22:22:48.055892 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:48.055722 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" Jul 27 22:22:48.055892 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:48.055731 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" event={"ID":"c66dd5b6-7e52-4d25-a3bb-4824049354b4","Type":"ContainerDied","Data":"c315d2ebd37b876d5e154268461546a292acd9b925bc2815772be4f056965f42"} Jul 27 22:22:48.055892 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:48.055784 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7" event={"ID":"c66dd5b6-7e52-4d25-a3bb-4824049354b4","Type":"ContainerDied","Data":"e9b05605548ecd727c6fb63b3253165a668c566677b3e765b343dd177833d00d"} Jul 27 22:22:48.055892 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:48.055806 2561 scope.go:117] "RemoveContainer" containerID="c315d2ebd37b876d5e154268461546a292acd9b925bc2815772be4f056965f42" Jul 27 22:22:48.067222 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:48.067194 2561 scope.go:117] "RemoveContainer" containerID="e5922dddd7e8ffbf1108f6871bbb6bacdd868b271cfb2b82e6a3020cba9122c2" Jul 27 22:22:48.075831 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:48.075810 2561 scope.go:117] "RemoveContainer" containerID="c315d2ebd37b876d5e154268461546a292acd9b925bc2815772be4f056965f42" Jul 27 22:22:48.076096 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:22:48.076075 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c315d2ebd37b876d5e154268461546a292acd9b925bc2815772be4f056965f42\": container with ID starting with c315d2ebd37b876d5e154268461546a292acd9b925bc2815772be4f056965f42 not found: ID does not exist" containerID="c315d2ebd37b876d5e154268461546a292acd9b925bc2815772be4f056965f42" Jul 27 22:22:48.076172 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:48.076110 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c315d2ebd37b876d5e154268461546a292acd9b925bc2815772be4f056965f42"} err="failed to get container status \"c315d2ebd37b876d5e154268461546a292acd9b925bc2815772be4f056965f42\": rpc error: code = NotFound desc = could not find container \"c315d2ebd37b876d5e154268461546a292acd9b925bc2815772be4f056965f42\": container with ID starting with c315d2ebd37b876d5e154268461546a292acd9b925bc2815772be4f056965f42 not found: ID does not exist" Jul 27 22:22:48.076172 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:48.076135 2561 scope.go:117] "RemoveContainer" containerID="e5922dddd7e8ffbf1108f6871bbb6bacdd868b271cfb2b82e6a3020cba9122c2" Jul 27 22:22:48.076429 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:22:48.076411 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5922dddd7e8ffbf1108f6871bbb6bacdd868b271cfb2b82e6a3020cba9122c2\": container with ID starting with e5922dddd7e8ffbf1108f6871bbb6bacdd868b271cfb2b82e6a3020cba9122c2 not found: ID does not exist" containerID="e5922dddd7e8ffbf1108f6871bbb6bacdd868b271cfb2b82e6a3020cba9122c2" Jul 27 22:22:48.076473 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:48.076436 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5922dddd7e8ffbf1108f6871bbb6bacdd868b271cfb2b82e6a3020cba9122c2"} err="failed to get container status \"e5922dddd7e8ffbf1108f6871bbb6bacdd868b271cfb2b82e6a3020cba9122c2\": rpc error: code = NotFound desc = could not find container \"e5922dddd7e8ffbf1108f6871bbb6bacdd868b271cfb2b82e6a3020cba9122c2\": container with ID starting with e5922dddd7e8ffbf1108f6871bbb6bacdd868b271cfb2b82e6a3020cba9122c2 not found: ID does not exist" Jul 27 22:22:48.086406 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:48.086377 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7"] Jul 27 22:22:48.088530 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:48.088492 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-00544d-openshift-default-659b7d8466-kskx7"] Jul 27 22:22:48.817990 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:48.817956 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" path="/var/lib/kubelet/pods/c66dd5b6-7e52-4d25-a3bb-4824049354b4/volumes" Jul 27 22:22:50.731204 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:50.731169 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:51.288847 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.288817 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:51.617983 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.617911 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx"] Jul 27 22:22:51.618242 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.618230 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1069ccba-9c2a-434d-b832-6d460e231ffc" containerName="istio-proxy" Jul 27 22:22:51.618302 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.618243 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1069ccba-9c2a-434d-b832-6d460e231ffc" containerName="istio-proxy" Jul 27 22:22:51.618302 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.618251 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerName="istio-proxy" Jul 27 22:22:51.618302 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.618257 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerName="istio-proxy" Jul 27 22:22:51.618302 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.618281 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerName="istio-proxy" Jul 27 22:22:51.618302 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.618290 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerName="istio-proxy" Jul 27 22:22:51.618452 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.618353 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1069ccba-9c2a-434d-b832-6d460e231ffc" containerName="istio-proxy" Jul 27 22:22:51.618452 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.618361 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerName="istio-proxy" Jul 27 22:22:51.618452 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.618371 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66dd5b6-7e52-4d25-a3bb-4824049354b4" containerName="istio-proxy" Jul 27 22:22:51.622624 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.622597 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.625411 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.625386 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c75b14-openshift-default-dockercfg-xzcst\"" Jul 27 22:22:51.643848 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.643818 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx"] Jul 27 22:22:51.656091 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.656064 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:51.792929 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.792894 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zhf9\" (UniqueName: \"kubernetes.io/projected/1ce3c12e-b7eb-4be2-8798-f302c39062e7-kube-api-access-7zhf9\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.793376 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.792945 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-podinfo\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.793376 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.792975 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-token\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.793376 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.793001 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istiod-ca-cert\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.793376 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.793027 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-workload-socket\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.793376 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.793103 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-envoy\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.793376 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.793126 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-workload-certs\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.793376 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.793158 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-data\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.793376 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.793190 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-credential-socket\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.894512 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.894428 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-podinfo\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.894512 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.894464 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-token\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.894512 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.894499 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istiod-ca-cert\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.894835 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.894540 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-workload-socket\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.894835 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.894571 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-envoy\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.894835 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.894598 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-workload-certs\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.894835 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.894634 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-data\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.894835 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.894680 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-credential-socket\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.894835 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.894759 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zhf9\" (UniqueName: \"kubernetes.io/projected/1ce3c12e-b7eb-4be2-8798-f302c39062e7-kube-api-access-7zhf9\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.895147 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.894994 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-workload-socket\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.895147 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.895083 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-data\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.895255 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.895201 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-credential-socket\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.895642 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.895610 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-workload-certs\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.895774 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.895648 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istiod-ca-cert\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.896980 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.896959 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-envoy\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.897106 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.897091 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-podinfo\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.910130 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.910103 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zhf9\" (UniqueName: \"kubernetes.io/projected/1ce3c12e-b7eb-4be2-8798-f302c39062e7-kube-api-access-7zhf9\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.910255 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.910142 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-token\") pod \"e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:51.934201 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:51.934161 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:52.091440 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:52.091411 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx"] Jul 27 22:22:52.093616 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:22:52.093589 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce3c12e_b7eb_4be2_8798_f302c39062e7.slice/crio-1d61c8a2fbdec9762d59891d3a4371ead0ec51aa5fcb34dfdec781517bfb99d0 WatchSource:0}: Error finding container 1d61c8a2fbdec9762d59891d3a4371ead0ec51aa5fcb34dfdec781517bfb99d0: Status 404 returned error can't find the container with id 1d61c8a2fbdec9762d59891d3a4371ead0ec51aa5fcb34dfdec781517bfb99d0 Jul 27 22:22:52.095908 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:52.095863 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:22:52.095997 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:52.095946 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:22:52.095997 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:52.095985 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:22:52.272433 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:52.272378 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:53.087165 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:53.087127 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" event={"ID":"1ce3c12e-b7eb-4be2-8798-f302c39062e7","Type":"ContainerStarted","Data":"8211dd6ee2fa7389a187192a9b29e229324ee28b8b09ee8298bb3a55c2f07589"} Jul 27 22:22:53.087603 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:53.087188 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" event={"ID":"1ce3c12e-b7eb-4be2-8798-f302c39062e7","Type":"ContainerStarted","Data":"1d61c8a2fbdec9762d59891d3a4371ead0ec51aa5fcb34dfdec781517bfb99d0"} Jul 27 22:22:53.113818 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:53.113765 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" podStartSLOduration=2.113747761 podStartE2EDuration="2.113747761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:22:53.11277681 +0000 UTC m=+2250.730214129" watchObservedRunningTime="2026-07-27 22:22:53.113747761 +0000 UTC m=+2250.731185080" Jul 27 22:22:53.935314 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:53.935278 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:53.952948 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:53.952913 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:54.091978 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:54.091947 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:54.092968 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:54.092946 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:22:55.263296 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:55.263253 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:55.365334 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:55.365251 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:55.509162 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:55.509128 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:55.774305 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:55.774248 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:56.117094 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:56.117003 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:56.218339 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:56.218302 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:59.815071 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:22:59.815034 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:00.041530 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:00.041446 2561 patch_prober.go:28] interesting pod/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 27 22:23:00.041530 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:00.041515 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" podUID="cf843206-c265-41bd-928d-c8e73d328ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 27 22:23:00.123639 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:00.123607 2561 generic.go:356] "Generic (PLEG): container finished" podID="cf843206-c265-41bd-928d-c8e73d328ac8" containerID="97c52d9a91eac04dee3944a47075f2106d006e450de88047de39ac77e44ca8ff" exitCode=0 Jul 27 22:23:00.123810 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:00.123686 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" event={"ID":"cf843206-c265-41bd-928d-c8e73d328ac8","Type":"ContainerDied","Data":"97c52d9a91eac04dee3944a47075f2106d006e450de88047de39ac77e44ca8ff"} Jul 27 22:23:00.124119 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:00.124106 2561 scope.go:117] "RemoveContainer" containerID="97c52d9a91eac04dee3944a47075f2106d006e450de88047de39ac77e44ca8ff" Jul 27 22:23:00.125245 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:00.125218 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:23:00.125364 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:00.125345 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:23:00.125434 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:00.125416 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:23:01.133052 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:01.133007 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" event={"ID":"cf843206-c265-41bd-928d-c8e73d328ac8","Type":"ContainerStarted","Data":"8f3ac3cc26330cc703bf5eb11bc42f7435c1f41642f4008f2acf89fe4c6a59e7"} Jul 27 22:23:01.491950 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:01.491902 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:23:01.492159 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:01.492128 2561 patch_prober.go:28] interesting pod/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 27 22:23:01.492237 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:01.492198 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" podUID="cf843206-c265-41bd-928d-c8e73d328ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 27 22:23:02.492158 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:02.492112 2561 patch_prober.go:28] interesting pod/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 27 22:23:02.492677 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:02.492182 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" podUID="cf843206-c265-41bd-928d-c8e73d328ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 27 22:23:03.492143 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:03.492096 2561 patch_prober.go:28] interesting pod/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 27 22:23:03.492373 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:03.492156 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" podUID="cf843206-c265-41bd-928d-c8e73d328ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 27 22:23:04.491533 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:04.491493 2561 patch_prober.go:28] interesting pod/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 27 22:23:04.491776 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:04.491554 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" podUID="cf843206-c265-41bd-928d-c8e73d328ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 27 22:23:05.495943 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:05.495904 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:23:05.496396 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:05.496372 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:23:05.497208 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:05.497182 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:23:07.325813 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:07.325771 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:07.388428 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:07.388386 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:09.094018 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:09.093976 2561 patch_prober.go:28] interesting pod/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 27 22:23:09.094468 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:09.094033 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" podUID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 27 22:23:09.186874 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:09.186843 2561 generic.go:356] "Generic (PLEG): container finished" podID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerID="8211dd6ee2fa7389a187192a9b29e229324ee28b8b09ee8298bb3a55c2f07589" exitCode=0 Jul 27 22:23:09.187055 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:09.186922 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" event={"ID":"1ce3c12e-b7eb-4be2-8798-f302c39062e7","Type":"ContainerDied","Data":"8211dd6ee2fa7389a187192a9b29e229324ee28b8b09ee8298bb3a55c2f07589"} Jul 27 22:23:09.187357 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:09.187339 2561 scope.go:117] "RemoveContainer" containerID="8211dd6ee2fa7389a187192a9b29e229324ee28b8b09ee8298bb3a55c2f07589" Jul 27 22:23:09.188478 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:09.188450 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:23:09.188575 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:09.188558 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:23:09.188616 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:09.188599 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:23:10.194470 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:10.194438 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" event={"ID":"1ce3c12e-b7eb-4be2-8798-f302c39062e7","Type":"ContainerStarted","Data":"9275de2378b45c0de0fa089e27fd7d4acde015e322103e7ba72ea0449ee43bd9"} Jul 27 22:23:10.935598 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:10.935558 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:23:10.935797 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:10.935630 2561 patch_prober.go:28] interesting pod/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 27 22:23:10.935797 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:10.935712 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" podUID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 27 22:23:11.935487 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:11.935444 2561 patch_prober.go:28] interesting pod/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 27 22:23:11.936001 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:11.935514 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" podUID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 27 22:23:12.935566 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:12.935527 2561 patch_prober.go:28] interesting pod/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 27 22:23:12.936209 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:12.936159 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" podUID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 27 22:23:13.934972 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:13.934939 2561 patch_prober.go:28] interesting pod/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 27 22:23:13.935164 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:13.934992 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" podUID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 27 22:23:14.939123 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:14.939092 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:23:14.939551 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:14.939375 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:23:14.940041 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:14.940025 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:23:23.815836 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:23.815800 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:23.894665 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:23.894629 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:23.967005 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:23.966968 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:28.261993 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:28.261952 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:28.676901 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:28.676866 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:35.484214 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:35.484181 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:35.560029 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:35.559993 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:35.765113 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:35.765036 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq"] Jul 27 22:23:35.765283 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:35.765246 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" podUID="cf843206-c265-41bd-928d-c8e73d328ac8" containerName="istio-proxy" containerID="cri-o://8f3ac3cc26330cc703bf5eb11bc42f7435c1f41642f4008f2acf89fe4c6a59e7" gracePeriod=30 Jul 27 22:23:35.790629 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:35.790568 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:36.013736 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:36.013701 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:41.023355 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.023327 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:23:41.155709 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.155675 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf843206-c265-41bd-928d-c8e73d328ac8-istiod-ca-cert\") pod \"cf843206-c265-41bd-928d-c8e73d328ac8\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " Jul 27 22:23:41.155869 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.155716 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf843206-c265-41bd-928d-c8e73d328ac8-istio-token\") pod \"cf843206-c265-41bd-928d-c8e73d328ac8\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " Jul 27 22:23:41.155869 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.155746 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-istio-data\") pod \"cf843206-c265-41bd-928d-c8e73d328ac8\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " Jul 27 22:23:41.155869 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.155767 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-workload-certs\") pod \"cf843206-c265-41bd-928d-c8e73d328ac8\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " Jul 27 22:23:41.155869 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.155788 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-credential-socket\") pod \"cf843206-c265-41bd-928d-c8e73d328ac8\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " Jul 27 22:23:41.155869 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.155838 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-istio-envoy\") pod \"cf843206-c265-41bd-928d-c8e73d328ac8\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " Jul 27 22:23:41.156112 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.155902 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-workload-socket\") pod \"cf843206-c265-41bd-928d-c8e73d328ac8\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " Jul 27 22:23:41.156112 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.155951 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnqht\" (UniqueName: \"kubernetes.io/projected/cf843206-c265-41bd-928d-c8e73d328ac8-kube-api-access-pnqht\") pod \"cf843206-c265-41bd-928d-c8e73d328ac8\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " Jul 27 22:23:41.156112 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.155990 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf843206-c265-41bd-928d-c8e73d328ac8-istio-podinfo\") pod \"cf843206-c265-41bd-928d-c8e73d328ac8\" (UID: \"cf843206-c265-41bd-928d-c8e73d328ac8\") " Jul 27 22:23:41.156112 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.156079 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf843206-c265-41bd-928d-c8e73d328ac8" (UID: "cf843206-c265-41bd-928d-c8e73d328ac8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:41.156357 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.156145 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf843206-c265-41bd-928d-c8e73d328ac8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf843206-c265-41bd-928d-c8e73d328ac8" (UID: "cf843206-c265-41bd-928d-c8e73d328ac8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:23:41.156357 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.156233 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf843206-c265-41bd-928d-c8e73d328ac8" (UID: "cf843206-c265-41bd-928d-c8e73d328ac8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:41.156357 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.156223 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf843206-c265-41bd-928d-c8e73d328ac8" (UID: "cf843206-c265-41bd-928d-c8e73d328ac8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:41.156357 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.156289 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.157130 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.157100 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf843206-c265-41bd-928d-c8e73d328ac8" (UID: "cf843206-c265-41bd-928d-c8e73d328ac8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:41.158130 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.158102 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf843206-c265-41bd-928d-c8e73d328ac8-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf843206-c265-41bd-928d-c8e73d328ac8" (UID: "cf843206-c265-41bd-928d-c8e73d328ac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:23:41.158230 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.158121 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf843206-c265-41bd-928d-c8e73d328ac8" (UID: "cf843206-c265-41bd-928d-c8e73d328ac8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:41.158230 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.158159 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf843206-c265-41bd-928d-c8e73d328ac8-kube-api-access-pnqht" (OuterVolumeSpecName: "kube-api-access-pnqht") pod "cf843206-c265-41bd-928d-c8e73d328ac8" (UID: "cf843206-c265-41bd-928d-c8e73d328ac8"). InnerVolumeSpecName "kube-api-access-pnqht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:23:41.158230 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.158176 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf843206-c265-41bd-928d-c8e73d328ac8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf843206-c265-41bd-928d-c8e73d328ac8" (UID: "cf843206-c265-41bd-928d-c8e73d328ac8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:23:41.256920 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.256879 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.256920 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.256915 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnqht\" (UniqueName: \"kubernetes.io/projected/cf843206-c265-41bd-928d-c8e73d328ac8-kube-api-access-pnqht\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.256920 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.256925 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf843206-c265-41bd-928d-c8e73d328ac8-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.257165 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.256934 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf843206-c265-41bd-928d-c8e73d328ac8-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.257165 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.256942 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf843206-c265-41bd-928d-c8e73d328ac8-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.257165 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.256950 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.257165 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.256957 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.257165 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.256966 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf843206-c265-41bd-928d-c8e73d328ac8-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.358663 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.358627 2561 generic.go:356] "Generic (PLEG): container finished" podID="cf843206-c265-41bd-928d-c8e73d328ac8" containerID="8f3ac3cc26330cc703bf5eb11bc42f7435c1f41642f4008f2acf89fe4c6a59e7" exitCode=0 Jul 27 22:23:41.358827 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.358681 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" Jul 27 22:23:41.358827 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.358714 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" event={"ID":"cf843206-c265-41bd-928d-c8e73d328ac8","Type":"ContainerDied","Data":"8f3ac3cc26330cc703bf5eb11bc42f7435c1f41642f4008f2acf89fe4c6a59e7"} Jul 27 22:23:41.358827 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.358768 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq" event={"ID":"cf843206-c265-41bd-928d-c8e73d328ac8","Type":"ContainerDied","Data":"bf5ec698c197aa44221ce0405055e96c1140d4895ae0cf06557fd3fd55b9d565"} Jul 27 22:23:41.358827 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.358791 2561 scope.go:117] "RemoveContainer" containerID="8f3ac3cc26330cc703bf5eb11bc42f7435c1f41642f4008f2acf89fe4c6a59e7" Jul 27 22:23:41.367753 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.367734 2561 scope.go:117] "RemoveContainer" containerID="97c52d9a91eac04dee3944a47075f2106d006e450de88047de39ac77e44ca8ff" Jul 27 22:23:41.377680 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.377662 2561 scope.go:117] "RemoveContainer" containerID="8f3ac3cc26330cc703bf5eb11bc42f7435c1f41642f4008f2acf89fe4c6a59e7" Jul 27 22:23:41.378217 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:23:41.377997 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3ac3cc26330cc703bf5eb11bc42f7435c1f41642f4008f2acf89fe4c6a59e7\": container with ID starting with 8f3ac3cc26330cc703bf5eb11bc42f7435c1f41642f4008f2acf89fe4c6a59e7 not found: ID does not exist" containerID="8f3ac3cc26330cc703bf5eb11bc42f7435c1f41642f4008f2acf89fe4c6a59e7" Jul 27 22:23:41.378217 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.378028 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3ac3cc26330cc703bf5eb11bc42f7435c1f41642f4008f2acf89fe4c6a59e7"} err="failed to get container status \"8f3ac3cc26330cc703bf5eb11bc42f7435c1f41642f4008f2acf89fe4c6a59e7\": rpc error: code = NotFound desc = could not find container \"8f3ac3cc26330cc703bf5eb11bc42f7435c1f41642f4008f2acf89fe4c6a59e7\": container with ID starting with 8f3ac3cc26330cc703bf5eb11bc42f7435c1f41642f4008f2acf89fe4c6a59e7 not found: ID does not exist" Jul 27 22:23:41.378217 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.378051 2561 scope.go:117] "RemoveContainer" containerID="97c52d9a91eac04dee3944a47075f2106d006e450de88047de39ac77e44ca8ff" Jul 27 22:23:41.378456 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:23:41.378346 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c52d9a91eac04dee3944a47075f2106d006e450de88047de39ac77e44ca8ff\": container with ID starting with 97c52d9a91eac04dee3944a47075f2106d006e450de88047de39ac77e44ca8ff not found: ID does not exist" containerID="97c52d9a91eac04dee3944a47075f2106d006e450de88047de39ac77e44ca8ff" Jul 27 22:23:41.378456 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.378374 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c52d9a91eac04dee3944a47075f2106d006e450de88047de39ac77e44ca8ff"} err="failed to get container status \"97c52d9a91eac04dee3944a47075f2106d006e450de88047de39ac77e44ca8ff\": rpc error: code = NotFound desc = could not find container \"97c52d9a91eac04dee3944a47075f2106d006e450de88047de39ac77e44ca8ff\": container with ID starting with 97c52d9a91eac04dee3944a47075f2106d006e450de88047de39ac77e44ca8ff not found: ID does not exist" Jul 27 22:23:41.380908 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.380887 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq"] Jul 27 22:23:41.387855 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:41.387833 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-721ee8-openshift-default-56db875588-r8gdq"] Jul 27 22:23:42.819913 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:42.819879 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf843206-c265-41bd-928d-c8e73d328ac8" path="/var/lib/kubelet/pods/cf843206-c265-41bd-928d-c8e73d328ac8/volumes" Jul 27 22:23:51.719716 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:51.719670 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:51.820111 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:51.820076 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:56.216887 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:56.216839 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:56.613004 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:23:56.612898 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:02.456152 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:02.456113 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:02.519091 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:02.519057 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:02.711605 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:02.711520 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx"] Jul 27 22:24:02.711780 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:02.711706 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" podUID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerName="istio-proxy" containerID="cri-o://9275de2378b45c0de0fa089e27fd7d4acde015e322103e7ba72ea0449ee43bd9" gracePeriod=30 Jul 27 22:24:02.732520 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:02.732478 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:02.972985 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:02.972890 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:03.362422 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.362335 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:03.658742 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.658708 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8"] Jul 27 22:24:03.659085 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.659033 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf843206-c265-41bd-928d-c8e73d328ac8" containerName="istio-proxy" Jul 27 22:24:03.659085 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.659043 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf843206-c265-41bd-928d-c8e73d328ac8" containerName="istio-proxy" Jul 27 22:24:03.659172 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.659127 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf843206-c265-41bd-928d-c8e73d328ac8" containerName="istio-proxy" Jul 27 22:24:03.662301 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.662283 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.664649 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.664630 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-6138cc-openshift-default-dockercfg-snk9t\"" Jul 27 22:24:03.676763 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.676740 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8"] Jul 27 22:24:03.690050 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.690000 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:03.737959 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.737919 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43d75b4-e488-448d-a196-50eac2d9a087-istio-token\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.738144 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.737966 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-workload-certs\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.738144 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.738017 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43d75b4-e488-448d-a196-50eac2d9a087-istio-podinfo\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.738144 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.738074 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-credential-socket\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.738144 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.738111 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-istio-envoy\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.738358 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.738165 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpv4v\" (UniqueName: \"kubernetes.io/projected/d43d75b4-e488-448d-a196-50eac2d9a087-kube-api-access-lpv4v\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.738358 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.738189 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-istio-data\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.738358 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.738205 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43d75b4-e488-448d-a196-50eac2d9a087-istiod-ca-cert\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.738358 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.738289 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-workload-socket\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.839417 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.839372 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-istio-data\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.839417 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.839418 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43d75b4-e488-448d-a196-50eac2d9a087-istiod-ca-cert\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.839650 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.839461 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-workload-socket\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.839650 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.839499 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43d75b4-e488-448d-a196-50eac2d9a087-istio-token\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.839650 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.839532 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-workload-certs\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.839650 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.839553 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43d75b4-e488-448d-a196-50eac2d9a087-istio-podinfo\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.839650 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.839606 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-credential-socket\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.839927 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.839661 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-istio-envoy\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.839927 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.839712 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpv4v\" (UniqueName: \"kubernetes.io/projected/d43d75b4-e488-448d-a196-50eac2d9a087-kube-api-access-lpv4v\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.839927 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.839828 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-istio-data\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.839927 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.839878 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-workload-certs\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.840140 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.840007 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-workload-socket\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.840140 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.840063 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-credential-socket\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.840403 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.840386 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43d75b4-e488-448d-a196-50eac2d9a087-istiod-ca-cert\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.841924 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.841900 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-istio-envoy\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.842060 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.842042 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43d75b4-e488-448d-a196-50eac2d9a087-istio-podinfo\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.845780 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.845753 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:03.847444 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.847420 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43d75b4-e488-448d-a196-50eac2d9a087-istio-token\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.847682 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.847667 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpv4v\" (UniqueName: \"kubernetes.io/projected/d43d75b4-e488-448d-a196-50eac2d9a087-kube-api-access-lpv4v\") pod \"e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:03.972794 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:03.972698 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:04.113357 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:04.113328 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8"] Jul 27 22:24:04.115730 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:24:04.115695 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43d75b4_e488_448d_a196_50eac2d9a087.slice/crio-1fd4186bfeddb0805b299495e1581057e715323723f7391f554e1ef6ec34a11d WatchSource:0}: Error finding container 1fd4186bfeddb0805b299495e1581057e715323723f7391f554e1ef6ec34a11d: Status 404 returned error can't find the container with id 1fd4186bfeddb0805b299495e1581057e715323723f7391f554e1ef6ec34a11d Jul 27 22:24:04.117912 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:04.117884 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:24:04.117995 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:04.117947 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:24:04.117995 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:04.117977 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:24:04.479856 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:04.479821 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" event={"ID":"d43d75b4-e488-448d-a196-50eac2d9a087","Type":"ContainerStarted","Data":"0abe9e9f037d6ad1923f74f52d2de41661ceb4082d44b0626b5898b23fb5ea5e"} Jul 27 22:24:04.480043 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:04.479875 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" event={"ID":"d43d75b4-e488-448d-a196-50eac2d9a087","Type":"ContainerStarted","Data":"1fd4186bfeddb0805b299495e1581057e715323723f7391f554e1ef6ec34a11d"} Jul 27 22:24:04.501979 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:04.501913 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" podStartSLOduration=1.501893702 podStartE2EDuration="1.501893702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:24:04.501235392 +0000 UTC m=+2322.118672711" watchObservedRunningTime="2026-07-27 22:24:04.501893702 +0000 UTC m=+2322.119331022" Jul 27 22:24:04.972806 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:04.972771 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:05.978147 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:05.978119 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:06.481926 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:06.481890 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:06.495876 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:06.495845 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:06.496767 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:06.496746 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:06.574129 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:06.574086 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:06.650614 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:06.650563 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:06.835046 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:06.834952 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:07.207032 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.206991 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:07.311017 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.310984 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:07.959244 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.959220 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:24:07.973242 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973167 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-token\") pod \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " Jul 27 22:24:07.973242 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973202 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-workload-socket\") pod \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " Jul 27 22:24:07.973242 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973243 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zhf9\" (UniqueName: \"kubernetes.io/projected/1ce3c12e-b7eb-4be2-8798-f302c39062e7-kube-api-access-7zhf9\") pod \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " Jul 27 22:24:07.973525 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973282 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-podinfo\") pod \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " Jul 27 22:24:07.973525 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973313 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-envoy\") pod \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " Jul 27 22:24:07.973525 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973344 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-workload-certs\") pod \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " Jul 27 22:24:07.973525 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973382 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-credential-socket\") pod \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " Jul 27 22:24:07.973525 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973447 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istiod-ca-cert\") pod \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " Jul 27 22:24:07.973525 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973491 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-data\") pod \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\" (UID: \"1ce3c12e-b7eb-4be2-8798-f302c39062e7\") " Jul 27 22:24:07.973839 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973580 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ce3c12e-b7eb-4be2-8798-f302c39062e7" (UID: "1ce3c12e-b7eb-4be2-8798-f302c39062e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:07.973839 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973745 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ce3c12e-b7eb-4be2-8798-f302c39062e7" (UID: "1ce3c12e-b7eb-4be2-8798-f302c39062e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:07.973953 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973855 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:24:07.973953 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973874 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:24:07.974078 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.974031 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ce3c12e-b7eb-4be2-8798-f302c39062e7" (UID: "1ce3c12e-b7eb-4be2-8798-f302c39062e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:07.974078 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.973963 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ce3c12e-b7eb-4be2-8798-f302c39062e7" (UID: "1ce3c12e-b7eb-4be2-8798-f302c39062e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:24:07.974601 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.974563 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ce3c12e-b7eb-4be2-8798-f302c39062e7" (UID: "1ce3c12e-b7eb-4be2-8798-f302c39062e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:07.975616 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.975589 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ce3c12e-b7eb-4be2-8798-f302c39062e7" (UID: "1ce3c12e-b7eb-4be2-8798-f302c39062e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:07.975720 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.975680 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ce3c12e-b7eb-4be2-8798-f302c39062e7" (UID: "1ce3c12e-b7eb-4be2-8798-f302c39062e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:24:07.975720 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.975690 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce3c12e-b7eb-4be2-8798-f302c39062e7-kube-api-access-7zhf9" (OuterVolumeSpecName: "kube-api-access-7zhf9") pod "1ce3c12e-b7eb-4be2-8798-f302c39062e7" (UID: "1ce3c12e-b7eb-4be2-8798-f302c39062e7"). InnerVolumeSpecName "kube-api-access-7zhf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:24:07.975800 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:07.975731 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ce3c12e-b7eb-4be2-8798-f302c39062e7" (UID: "1ce3c12e-b7eb-4be2-8798-f302c39062e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:24:08.074866 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.074834 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zhf9\" (UniqueName: \"kubernetes.io/projected/1ce3c12e-b7eb-4be2-8798-f302c39062e7-kube-api-access-7zhf9\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:24:08.074866 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.074862 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:24:08.074866 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.074872 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:24:08.075103 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.074881 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:24:08.075103 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.074889 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:24:08.075103 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.074899 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:24:08.075103 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.074910 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce3c12e-b7eb-4be2-8798-f302c39062e7-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:24:08.508646 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.508619 2561 generic.go:356] "Generic (PLEG): container finished" podID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerID="9275de2378b45c0de0fa089e27fd7d4acde015e322103e7ba72ea0449ee43bd9" exitCode=0 Jul 27 22:24:08.509059 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.508666 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" Jul 27 22:24:08.509059 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.508703 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" event={"ID":"1ce3c12e-b7eb-4be2-8798-f302c39062e7","Type":"ContainerDied","Data":"9275de2378b45c0de0fa089e27fd7d4acde015e322103e7ba72ea0449ee43bd9"} Jul 27 22:24:08.509059 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.508764 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx" event={"ID":"1ce3c12e-b7eb-4be2-8798-f302c39062e7","Type":"ContainerDied","Data":"1d61c8a2fbdec9762d59891d3a4371ead0ec51aa5fcb34dfdec781517bfb99d0"} Jul 27 22:24:08.509059 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.508789 2561 scope.go:117] "RemoveContainer" containerID="9275de2378b45c0de0fa089e27fd7d4acde015e322103e7ba72ea0449ee43bd9" Jul 27 22:24:08.517939 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.517923 2561 scope.go:117] "RemoveContainer" containerID="8211dd6ee2fa7389a187192a9b29e229324ee28b8b09ee8298bb3a55c2f07589" Jul 27 22:24:08.525808 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.525792 2561 scope.go:117] "RemoveContainer" containerID="9275de2378b45c0de0fa089e27fd7d4acde015e322103e7ba72ea0449ee43bd9" Jul 27 22:24:08.526036 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:24:08.526020 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9275de2378b45c0de0fa089e27fd7d4acde015e322103e7ba72ea0449ee43bd9\": container with ID starting with 9275de2378b45c0de0fa089e27fd7d4acde015e322103e7ba72ea0449ee43bd9 not found: ID does not exist" containerID="9275de2378b45c0de0fa089e27fd7d4acde015e322103e7ba72ea0449ee43bd9" Jul 27 22:24:08.526095 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.526043 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9275de2378b45c0de0fa089e27fd7d4acde015e322103e7ba72ea0449ee43bd9"} err="failed to get container status \"9275de2378b45c0de0fa089e27fd7d4acde015e322103e7ba72ea0449ee43bd9\": rpc error: code = NotFound desc = could not find container \"9275de2378b45c0de0fa089e27fd7d4acde015e322103e7ba72ea0449ee43bd9\": container with ID starting with 9275de2378b45c0de0fa089e27fd7d4acde015e322103e7ba72ea0449ee43bd9 not found: ID does not exist" Jul 27 22:24:08.526095 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.526059 2561 scope.go:117] "RemoveContainer" containerID="8211dd6ee2fa7389a187192a9b29e229324ee28b8b09ee8298bb3a55c2f07589" Jul 27 22:24:08.526271 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:24:08.526245 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8211dd6ee2fa7389a187192a9b29e229324ee28b8b09ee8298bb3a55c2f07589\": container with ID starting with 8211dd6ee2fa7389a187192a9b29e229324ee28b8b09ee8298bb3a55c2f07589 not found: ID does not exist" containerID="8211dd6ee2fa7389a187192a9b29e229324ee28b8b09ee8298bb3a55c2f07589" Jul 27 22:24:08.526324 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.526276 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8211dd6ee2fa7389a187192a9b29e229324ee28b8b09ee8298bb3a55c2f07589"} err="failed to get container status \"8211dd6ee2fa7389a187192a9b29e229324ee28b8b09ee8298bb3a55c2f07589\": rpc error: code = NotFound desc = could not find container \"8211dd6ee2fa7389a187192a9b29e229324ee28b8b09ee8298bb3a55c2f07589\": container with ID starting with 8211dd6ee2fa7389a187192a9b29e229324ee28b8b09ee8298bb3a55c2f07589 not found: ID does not exist" Jul 27 22:24:08.530099 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.530076 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx"] Jul 27 22:24:08.535981 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.535962 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c75b14-openshift-default-76788958f4-wr9hx"] Jul 27 22:24:08.818553 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:08.818473 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" path="/var/lib/kubelet/pods/1ce3c12e-b7eb-4be2-8798-f302c39062e7/volumes" Jul 27 22:24:11.021032 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.021002 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:11.347153 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.347064 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:11.741956 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.741922 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp"] Jul 27 22:24:11.742325 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.742298 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerName="istio-proxy" Jul 27 22:24:11.742325 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.742320 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerName="istio-proxy" Jul 27 22:24:11.742533 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.742345 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf843206-c265-41bd-928d-c8e73d328ac8" containerName="istio-proxy" Jul 27 22:24:11.742533 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.742351 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf843206-c265-41bd-928d-c8e73d328ac8" containerName="istio-proxy" Jul 27 22:24:11.742533 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.742415 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf843206-c265-41bd-928d-c8e73d328ac8" containerName="istio-proxy" Jul 27 22:24:11.742533 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.742432 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerName="istio-proxy" Jul 27 22:24:11.742533 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.742439 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerName="istio-proxy" Jul 27 22:24:11.746956 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.746938 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.749330 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.749309 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-0c79f1-openshift-default-dockercfg-64n2f\"" Jul 27 22:24:11.757345 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.757322 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp"] Jul 27 22:24:11.780542 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.780479 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:11.808285 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.808242 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-credential-socket\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.808461 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.808296 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-istio-envoy\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.808461 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.808411 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-workload-certs\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.808546 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.808466 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b18b192-a462-4649-9416-c411c15ad5f2-istio-podinfo\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.808546 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.808498 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-workload-socket\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.808546 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.808526 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b18b192-a462-4649-9416-c411c15ad5f2-istiod-ca-cert\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.808642 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.808583 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6w25\" (UniqueName: \"kubernetes.io/projected/9b18b192-a462-4649-9416-c411c15ad5f2-kube-api-access-c6w25\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.808673 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.808644 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-istio-data\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.808704 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.808676 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b18b192-a462-4649-9416-c411c15ad5f2-istio-token\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.909839 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.909805 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-workload-certs\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.910109 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.909851 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b18b192-a462-4649-9416-c411c15ad5f2-istio-podinfo\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.910109 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.909870 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-workload-socket\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.910109 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.909919 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b18b192-a462-4649-9416-c411c15ad5f2-istiod-ca-cert\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.910109 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.909964 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6w25\" (UniqueName: \"kubernetes.io/projected/9b18b192-a462-4649-9416-c411c15ad5f2-kube-api-access-c6w25\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.910109 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.910017 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-istio-data\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.910109 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.910054 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b18b192-a462-4649-9416-c411c15ad5f2-istio-token\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.910109 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.910095 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-credential-socket\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.910551 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.910133 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-istio-envoy\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.910551 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.910296 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-workload-socket\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.910551 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.910394 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-istio-data\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.910688 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.910563 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-credential-socket\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.910792 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.910760 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b18b192-a462-4649-9416-c411c15ad5f2-istiod-ca-cert\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.910920 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.910873 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-workload-certs\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.912501 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.912479 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-istio-envoy\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.912600 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.912564 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b18b192-a462-4649-9416-c411c15ad5f2-istio-podinfo\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.919821 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.919794 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b18b192-a462-4649-9416-c411c15ad5f2-istio-token\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.926428 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.926395 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6w25\" (UniqueName: \"kubernetes.io/projected/9b18b192-a462-4649-9416-c411c15ad5f2-kube-api-access-c6w25\") pod \"e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:11.973573 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:11.973521 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:12.058018 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:12.057925 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:12.194892 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:12.194865 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp"] Jul 27 22:24:12.196776 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:24:12.196751 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b18b192_a462_4649_9416_c411c15ad5f2.slice/crio-e8f5707a1c6d1b7ba064ce12a84bba58140e9f869f3d14b9722a419785c32eca WatchSource:0}: Error finding container e8f5707a1c6d1b7ba064ce12a84bba58140e9f869f3d14b9722a419785c32eca: Status 404 returned error can't find the container with id e8f5707a1c6d1b7ba064ce12a84bba58140e9f869f3d14b9722a419785c32eca Jul 27 22:24:12.199001 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:12.198961 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:24:12.199077 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:12.199051 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:24:12.199115 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:12.199091 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:24:12.530738 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:12.530703 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" event={"ID":"9b18b192-a462-4649-9416-c411c15ad5f2","Type":"ContainerStarted","Data":"2bad1e735e4361b48e692746d4c6b0f7fb6b4814b0c4d8a9ee4fa4d1fef45d70"} Jul 27 22:24:12.530924 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:12.530754 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" event={"ID":"9b18b192-a462-4649-9416-c411c15ad5f2","Type":"ContainerStarted","Data":"e8f5707a1c6d1b7ba064ce12a84bba58140e9f869f3d14b9722a419785c32eca"} Jul 27 22:24:12.552570 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:12.552518 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" podStartSLOduration=1.552497491 podStartE2EDuration="1.552497491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:24:12.549880805 +0000 UTC m=+2330.167318124" watchObservedRunningTime="2026-07-27 22:24:12.552497491 +0000 UTC m=+2330.169934809" Jul 27 22:24:13.058966 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:13.058925 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:13.063929 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:13.063906 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:13.536225 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:13.536199 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:13.537196 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:13.537179 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:14.531020 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:14.530926 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:14.646112 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:14.646067 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:14.737800 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:14.737766 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:15.115512 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:15.115476 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:15.224988 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:15.224951 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:15.325039 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:15.324987 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:19.064469 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:19.064368 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:21.631154 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:21.631110 2561 patch_prober.go:28] interesting pod/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:44150->10.133.0.57:15021: read: connection reset by peer" start-of-body= Jul 27 22:24:21.633667 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:21.631186 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" podUID="d43d75b4-e488-448d-a196-50eac2d9a087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:44150->10.133.0.57:15021: read: connection reset by peer" Jul 27 22:24:22.595017 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:22.594981 2561 generic.go:356] "Generic (PLEG): container finished" podID="d43d75b4-e488-448d-a196-50eac2d9a087" containerID="0abe9e9f037d6ad1923f74f52d2de41661ceb4082d44b0626b5898b23fb5ea5e" exitCode=0 Jul 27 22:24:22.595210 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:22.595058 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" event={"ID":"d43d75b4-e488-448d-a196-50eac2d9a087","Type":"ContainerDied","Data":"0abe9e9f037d6ad1923f74f52d2de41661ceb4082d44b0626b5898b23fb5ea5e"} Jul 27 22:24:22.595517 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:22.595501 2561 scope.go:117] "RemoveContainer" containerID="0abe9e9f037d6ad1923f74f52d2de41661ceb4082d44b0626b5898b23fb5ea5e" Jul 27 22:24:22.596559 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:22.596535 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:24:22.596672 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:22.596596 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:24:22.596672 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:22.596627 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:24:23.093233 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:23.093200 2561 scope.go:117] "RemoveContainer" containerID="85dbc9a648e3ef4373c87137543bdc0fba05993eb7da5940b9b40d5538e8b9ed" Jul 27 22:24:23.603596 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:23.603560 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" event={"ID":"d43d75b4-e488-448d-a196-50eac2d9a087","Type":"ContainerStarted","Data":"b98d0a1d3c4602137353e53610e91791553ab5447db531350f96e8a49047ce3c"} Jul 27 22:24:23.973436 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:23.973401 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:23.973819 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:23.973793 2561 patch_prober.go:28] interesting pod/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 27 22:24:23.974010 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:23.973984 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" podUID="d43d75b4-e488-448d-a196-50eac2d9a087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 27 22:24:24.973361 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:24.973318 2561 patch_prober.go:28] interesting pod/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 27 22:24:24.973981 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:24.973947 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" podUID="d43d75b4-e488-448d-a196-50eac2d9a087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 27 22:24:25.974178 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:25.974139 2561 patch_prober.go:28] interesting pod/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 27 22:24:25.974606 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:25.974202 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" podUID="d43d75b4-e488-448d-a196-50eac2d9a087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 27 22:24:26.973457 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:26.973415 2561 patch_prober.go:28] interesting pod/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 27 22:24:26.973631 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:26.973479 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" podUID="d43d75b4-e488-448d-a196-50eac2d9a087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 27 22:24:27.976923 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:27.976890 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:27.977344 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:27.977228 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:27.977939 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:27.977919 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:24:28.537963 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:28.537932 2561 patch_prober.go:28] interesting pod/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 27 22:24:28.538168 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:28.537984 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 27 22:24:28.633083 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:28.633006 2561 generic.go:356] "Generic (PLEG): container finished" podID="9b18b192-a462-4649-9416-c411c15ad5f2" containerID="2bad1e735e4361b48e692746d4c6b0f7fb6b4814b0c4d8a9ee4fa4d1fef45d70" exitCode=0 Jul 27 22:24:28.633237 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:28.633080 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" event={"ID":"9b18b192-a462-4649-9416-c411c15ad5f2","Type":"ContainerDied","Data":"2bad1e735e4361b48e692746d4c6b0f7fb6b4814b0c4d8a9ee4fa4d1fef45d70"} Jul 27 22:24:28.633658 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:28.633636 2561 scope.go:117] "RemoveContainer" containerID="2bad1e735e4361b48e692746d4c6b0f7fb6b4814b0c4d8a9ee4fa4d1fef45d70" Jul 27 22:24:28.634596 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:28.634571 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:24:28.634686 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:28.634640 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:24:28.634739 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:28.634681 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:24:29.645684 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:29.645641 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" event={"ID":"9b18b192-a462-4649-9416-c411c15ad5f2","Type":"ContainerStarted","Data":"ef03af865fc4460331a804257a68dce2bc4979c151227686635de88626fab666"} Jul 27 22:24:30.058779 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:30.058735 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:30.059018 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:30.058993 2561 patch_prober.go:28] interesting pod/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 27 22:24:30.059081 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:30.059060 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 27 22:24:31.058618 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:31.058575 2561 patch_prober.go:28] interesting pod/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 27 22:24:31.059031 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:31.058649 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 27 22:24:32.058382 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:32.058341 2561 patch_prober.go:28] interesting pod/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 27 22:24:32.058570 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:32.058418 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 27 22:24:33.059354 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:33.059315 2561 patch_prober.go:28] interesting pod/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 27 22:24:33.059844 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:33.059379 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 27 22:24:34.062453 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:34.062422 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:34.062849 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:34.062696 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:34.063408 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:34.063392 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:24:37.986724 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:37.986684 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:52.115980 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:24:52.115943 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:13.112555 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:25:13.112519 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:13.203673 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:25:13.203637 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:24.436889 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:25:24.436849 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:52.321626 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:26:52.321586 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:55.858317 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:27:55.858280 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:57.030012 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:27:57.029977 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:37.955376 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:29:37.955333 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:38.864462 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:29:38.864425 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:50.886806 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:29:50.886758 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:25.762291 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:30:25.762187 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:41.256588 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:41.256548 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:41.358777 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:41.358739 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:41.923586 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:41.923552 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:42.769488 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:42.769459 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:54.657095 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.657060 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm"] Jul 27 22:31:54.657505 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.657442 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerName="istio-proxy" Jul 27 22:31:54.657505 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.657453 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ce3c12e-b7eb-4be2-8798-f302c39062e7" containerName="istio-proxy" Jul 27 22:31:54.660598 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.660582 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.663676 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.663650 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6138cc\"/\"openshift-service-ca.crt\"" Jul 27 22:31:54.663676 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.663666 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6138cc\"/\"default-dockercfg-jdcn5\"" Jul 27 22:31:54.663872 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.663650 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6138cc\"/\"rate-test-model-6138cc-kserve-self-signed-certs\"" Jul 27 22:31:54.663872 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.663701 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6138cc\"/\"kube-root-ca.crt\"" Jul 27 22:31:54.668982 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.668960 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm"] Jul 27 22:31:54.743517 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.743478 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9qvl\" (UniqueName: \"kubernetes.io/projected/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-kube-api-access-p9qvl\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.743517 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.743516 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-tmp-dir\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.743787 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.743562 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-home\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.743787 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.743597 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-dshm\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.743787 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.743632 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-model-cache\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.743787 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.743719 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-tls-certs\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.745168 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.745143 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:54.845009 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.844970 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-tls-certs\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.845214 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.845042 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9qvl\" (UniqueName: \"kubernetes.io/projected/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-kube-api-access-p9qvl\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.845669 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.845644 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-tmp-dir\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.845804 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.845251 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-tmp-dir\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.845872 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.845806 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-home\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.846127 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.846093 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-home\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.846127 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.845841 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-dshm\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.846546 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.846526 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-model-cache\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.847072 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.846965 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-model-cache\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.848495 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.848450 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-tls-certs\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.848821 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.848788 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-dshm\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.854455 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.854434 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9qvl\" (UniqueName: \"kubernetes.io/projected/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-kube-api-access-p9qvl\") pod \"rate-test-model-6138cc-kserve-777448bbff-xmcxm\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:54.971614 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:54.971517 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:31:55.108508 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:55.108481 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm"] Jul 27 22:31:55.110595 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:31:55.110567 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdb2d388_7bec_45ef_a62a_a5a2d6cf4c2d.slice/crio-9c2576b157f4163fbb2003180df4a6a6ebddddb899866d49145a63d9730984b5 WatchSource:0}: Error finding container 9c2576b157f4163fbb2003180df4a6a6ebddddb899866d49145a63d9730984b5: Status 404 returned error can't find the container with id 9c2576b157f4163fbb2003180df4a6a6ebddddb899866d49145a63d9730984b5 Jul 27 22:31:55.944846 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:55.944809 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" event={"ID":"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d","Type":"ContainerStarted","Data":"8be400aecb65d1f4960943e3c51305617594903e34af6f8fc62a2cc658536359"} Jul 27 22:31:55.945244 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:55.944858 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" event={"ID":"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d","Type":"ContainerStarted","Data":"9c2576b157f4163fbb2003180df4a6a6ebddddb899866d49145a63d9730984b5"} Jul 27 22:31:55.963774 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:31:55.963724 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" podStartSLOduration=1.963709277 podStartE2EDuration="1.963709277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:31:55.962399799 +0000 UTC m=+2793.579837143" watchObservedRunningTime="2026-07-27 22:31:55.963709277 +0000 UTC m=+2793.581146595" Jul 27 22:32:04.972336 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:32:04.972289 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:32:04.972336 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:32:04.972344 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:32:04.985026 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:32:04.984992 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:32:05.003553 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:32:05.003469 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:32:09.258491 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:32:09.258442 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:17.872920 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:32:17.872880 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:36.733213 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:36.733109 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm"] Jul 27 22:33:36.757753 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:36.757719 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:38.227620 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:38.227588 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:51.752319 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:51.752276 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" podUID="bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d" containerName="main" containerID="cri-o://8be400aecb65d1f4960943e3c51305617594903e34af6f8fc62a2cc658536359" gracePeriod=45 Jul 27 22:33:52.006732 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.006665 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:33:52.136289 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.136223 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9qvl\" (UniqueName: \"kubernetes.io/projected/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-kube-api-access-p9qvl\") pod \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " Jul 27 22:33:52.136289 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.136286 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-tls-certs\") pod \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " Jul 27 22:33:52.136549 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.136326 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-tmp-dir\") pod \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " Jul 27 22:33:52.136549 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.136340 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-dshm\") pod \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " Jul 27 22:33:52.136549 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.136367 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-model-cache\") pod \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " Jul 27 22:33:52.136549 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.136414 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-home\") pod \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\" (UID: \"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d\") " Jul 27 22:33:52.136798 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.136644 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d" (UID: "bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:52.136798 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.136701 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d" (UID: "bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:52.136798 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.136722 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-home" (OuterVolumeSpecName: "home") pod "bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d" (UID: "bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:52.136952 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.136803 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-home\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.136952 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.136820 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-tmp-dir\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.136952 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.136832 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-model-cache\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.138673 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.138648 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-dshm" (OuterVolumeSpecName: "dshm") pod "bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d" (UID: "bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:52.138790 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.138709 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d" (UID: "bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:33:52.138790 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.138711 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-kube-api-access-p9qvl" (OuterVolumeSpecName: "kube-api-access-p9qvl") pod "bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d" (UID: "bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d"). InnerVolumeSpecName "kube-api-access-p9qvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:33:52.238136 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.238104 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9qvl\" (UniqueName: \"kubernetes.io/projected/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-kube-api-access-p9qvl\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.238136 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.238132 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.238136 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.238143 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d-dshm\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.550396 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.550359 2561 generic.go:356] "Generic (PLEG): container finished" podID="bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d" containerID="8be400aecb65d1f4960943e3c51305617594903e34af6f8fc62a2cc658536359" exitCode=0 Jul 27 22:33:52.550579 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.550409 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" Jul 27 22:33:52.550579 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.550456 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" event={"ID":"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d","Type":"ContainerDied","Data":"8be400aecb65d1f4960943e3c51305617594903e34af6f8fc62a2cc658536359"} Jul 27 22:33:52.550579 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.550505 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm" event={"ID":"bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d","Type":"ContainerDied","Data":"9c2576b157f4163fbb2003180df4a6a6ebddddb899866d49145a63d9730984b5"} Jul 27 22:33:52.550579 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.550523 2561 scope.go:117] "RemoveContainer" containerID="8be400aecb65d1f4960943e3c51305617594903e34af6f8fc62a2cc658536359" Jul 27 22:33:52.559698 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.559672 2561 scope.go:117] "RemoveContainer" containerID="8be400aecb65d1f4960943e3c51305617594903e34af6f8fc62a2cc658536359" Jul 27 22:33:52.559970 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:33:52.559951 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be400aecb65d1f4960943e3c51305617594903e34af6f8fc62a2cc658536359\": container with ID starting with 8be400aecb65d1f4960943e3c51305617594903e34af6f8fc62a2cc658536359 not found: ID does not exist" containerID="8be400aecb65d1f4960943e3c51305617594903e34af6f8fc62a2cc658536359" Jul 27 22:33:52.560035 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.559979 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be400aecb65d1f4960943e3c51305617594903e34af6f8fc62a2cc658536359"} err="failed to get container status \"8be400aecb65d1f4960943e3c51305617594903e34af6f8fc62a2cc658536359\": rpc error: code = NotFound desc = could not find container \"8be400aecb65d1f4960943e3c51305617594903e34af6f8fc62a2cc658536359\": container with ID starting with 8be400aecb65d1f4960943e3c51305617594903e34af6f8fc62a2cc658536359 not found: ID does not exist" Jul 27 22:33:52.572315 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.572287 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm"] Jul 27 22:33:52.576645 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.576620 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/rate-test-model-6138cc-kserve-777448bbff-xmcxm"] Jul 27 22:33:52.820434 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:52.820353 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d" path="/var/lib/kubelet/pods/bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d/volumes" Jul 27 22:33:57.404229 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.404191 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:57.688024 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.687937 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6"] Jul 27 22:33:57.688505 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.688488 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d" containerName="main" Jul 27 22:33:57.688568 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.688507 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d" containerName="main" Jul 27 22:33:57.688620 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.688594 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb2d388-7bec-45ef-a62a-a5a2d6cf4c2d" containerName="main" Jul 27 22:33:57.692682 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.692660 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.696365 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.696344 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-626801-openshift-default-dockercfg-p4xdc\"" Jul 27 22:33:57.708564 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.708539 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6"] Jul 27 22:33:57.737314 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.737274 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:57.780555 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.780523 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-workload-socket\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.780777 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.780581 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-podinfo\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.780777 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.780623 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-token\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.780777 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.780653 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-workload-certs\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.780777 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.780697 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh98d\" (UniqueName: \"kubernetes.io/projected/501b57e9-af8c-423b-81c3-e745fe80e8aa-kube-api-access-vh98d\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.780777 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.780743 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-credential-socket\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.780777 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.780767 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-data\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.781102 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.780785 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-envoy\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.781102 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.780809 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501b57e9-af8c-423b-81c3-e745fe80e8aa-istiod-ca-cert\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.881859 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.881815 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-workload-socket\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.882074 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.881882 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-podinfo\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.882074 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.881906 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-token\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.882074 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.881922 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-workload-certs\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.882074 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.881946 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh98d\" (UniqueName: \"kubernetes.io/projected/501b57e9-af8c-423b-81c3-e745fe80e8aa-kube-api-access-vh98d\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.882332 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.882144 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-credential-socket\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.882332 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.882199 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-data\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.882332 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.882232 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-envoy\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.882332 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.882295 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501b57e9-af8c-423b-81c3-e745fe80e8aa-istiod-ca-cert\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.882332 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.882312 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-workload-certs\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.882601 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.882298 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-workload-socket\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.882601 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.882527 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-data\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.882601 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.882570 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-credential-socket\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.882952 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.882934 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501b57e9-af8c-423b-81c3-e745fe80e8aa-istiod-ca-cert\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.884413 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.884393 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-envoy\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.884628 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.884610 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-podinfo\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.890858 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.890830 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-token\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:57.891062 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:57.891042 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh98d\" (UniqueName: \"kubernetes.io/projected/501b57e9-af8c-423b-81c3-e745fe80e8aa-kube-api-access-vh98d\") pod \"e2e-ipp-a-626801-openshift-default-6789f4556-94sp6\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:58.003558 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:58.003466 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:58.101673 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:58.101635 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:58.154517 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:58.154490 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6"] Jul 27 22:33:58.156528 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:33:58.156499 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501b57e9_af8c_423b_81c3_e745fe80e8aa.slice/crio-561f839f21dac129753272fb8aa5333564ceee7cba6643849f65a69eb654d859 WatchSource:0}: Error finding container 561f839f21dac129753272fb8aa5333564ceee7cba6643849f65a69eb654d859: Status 404 returned error can't find the container with id 561f839f21dac129753272fb8aa5333564ceee7cba6643849f65a69eb654d859 Jul 27 22:33:58.159315 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:58.159250 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:33:58.159383 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:58.159353 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:33:58.159418 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:58.159380 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:33:58.585080 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:58.585035 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" event={"ID":"501b57e9-af8c-423b-81c3-e745fe80e8aa","Type":"ContainerStarted","Data":"c0ce02e60227e3012dae4019244c19369858a06bc68dbbe5dcb8e0bd96fa5862"} Jul 27 22:33:58.585080 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:58.585089 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" event={"ID":"501b57e9-af8c-423b-81c3-e745fe80e8aa","Type":"ContainerStarted","Data":"561f839f21dac129753272fb8aa5333564ceee7cba6643849f65a69eb654d859"} Jul 27 22:33:58.608236 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:58.608179 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" podStartSLOduration=1.608158448 podStartE2EDuration="1.608158448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:33:58.607701651 +0000 UTC m=+2916.225138971" watchObservedRunningTime="2026-07-27 22:33:58.608158448 +0000 UTC m=+2916.225595836" Jul 27 22:33:59.004068 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:59.004026 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:59.009414 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:59.009386 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:59.595361 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:59.593801 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:33:59.596321 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:33:59.596303 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:34:00.622543 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:00.622497 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:00.719511 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:00.719478 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:00.811628 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:00.811584 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:01.021594 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:01.021560 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:01.420657 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:01.420624 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:01.553576 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:01.553532 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:05.160996 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:05.160955 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:18.698456 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:18.698422 2561 generic.go:356] "Generic (PLEG): container finished" podID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerID="c0ce02e60227e3012dae4019244c19369858a06bc68dbbe5dcb8e0bd96fa5862" exitCode=0 Jul 27 22:34:18.698897 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:18.698490 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" event={"ID":"501b57e9-af8c-423b-81c3-e745fe80e8aa","Type":"ContainerDied","Data":"c0ce02e60227e3012dae4019244c19369858a06bc68dbbe5dcb8e0bd96fa5862"} Jul 27 22:34:18.698897 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:18.698863 2561 scope.go:117] "RemoveContainer" containerID="c0ce02e60227e3012dae4019244c19369858a06bc68dbbe5dcb8e0bd96fa5862" Jul 27 22:34:18.699865 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:18.699837 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:34:18.699983 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:18.699920 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:34:18.699983 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:18.699969 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 22:34:19.706308 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:19.706256 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" event={"ID":"501b57e9-af8c-423b-81c3-e745fe80e8aa","Type":"ContainerStarted","Data":"a721523f631b9efc3e459a23e58ad6f9bc070f60e07a797f791e417376546958"} Jul 27 22:34:20.003878 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:20.003775 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:34:20.004056 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:20.004034 2561 patch_prober.go:28] interesting pod/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 27 22:34:20.004119 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:20.004097 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 27 22:34:21.004530 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:21.004494 2561 patch_prober.go:28] interesting pod/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 27 22:34:21.004952 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:21.004571 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 27 22:34:21.256141 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:21.256054 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:21.577393 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:21.577234 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:21.713493 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:21.713462 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:22.004457 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:22.004420 2561 patch_prober.go:28] interesting pod/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 27 22:34:22.004655 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:22.004486 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 27 22:34:23.004369 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:23.004332 2561 patch_prober.go:28] interesting pod/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 27 22:34:23.004558 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:23.004391 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 27 22:34:24.004139 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:24.004102 2561 patch_prober.go:28] interesting pod/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 27 22:34:24.004558 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:24.004152 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 27 22:34:25.007707 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:25.007672 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:34:25.008165 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:25.008068 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:34:25.008682 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:25.008662 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:34:27.814191 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:27.814155 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:27.912277 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:27.912214 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:28.120183 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:28.120091 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:28.269439 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:28.269405 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:28.705734 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:28.705684 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:28.803347 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:28.803312 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:32.353374 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:32.353333 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:53.389023 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:34:53.388971 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:31.514120 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:31.514070 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:43.888047 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:43.888014 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:43.949504 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:43.949474 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:45.540120 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:45.540086 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:45.911905 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:45.911871 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:46.188728 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:46.188615 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:49.338242 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:49.338205 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:49.425193 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:49.425144 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:49.499016 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:49.498981 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:49.765183 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:49.765145 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:50.078177 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:50.078085 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:50.183852 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:50.183814 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:53.182639 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:53.182597 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:35:53.846693 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:35:53.846651 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:04.834481 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:04.834446 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c"] Jul 27 22:36:04.838112 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:04.838086 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c" Jul 27 22:36:04.838712 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:04.838687 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c"] Jul 27 22:36:04.840416 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:04.840390 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 22:36:04.840540 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:04.840439 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\"" Jul 27 22:36:04.874958 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:04.874924 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c\" (UID: \"6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c" Jul 27 22:36:04.976164 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:04.976124 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c\" (UID: \"6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c" Jul 27 22:36:04.976810 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:04.976788 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c\" (UID: \"6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c" Jul 27 22:36:05.150182 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:05.150152 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c" Jul 27 22:36:05.284123 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:05.284066 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c"] Jul 27 22:36:05.286468 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:36:05.286437 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2c6cb0_4e23_42b4_a8cb_b4d4ff7034cc.slice/crio-a30bf78ddcd8cb85dc09ea47eb7ff2da82850f54a8022bd629d399ecf839103f WatchSource:0}: Error finding container a30bf78ddcd8cb85dc09ea47eb7ff2da82850f54a8022bd629d399ecf839103f: Status 404 returned error can't find the container with id a30bf78ddcd8cb85dc09ea47eb7ff2da82850f54a8022bd629d399ecf839103f Jul 27 22:36:06.045718 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:06.045682 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4wrkw"] Jul 27 22:36:06.049012 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:06.048993 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" Jul 27 22:36:06.057684 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:06.057660 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4wrkw"] Jul 27 22:36:06.085893 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:06.085861 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlbkm\" (UniqueName: \"kubernetes.io/projected/e390b071-fae1-49c9-879e-be2226eee6a9-kube-api-access-zlbkm\") pod \"kuadrant-operator-catalog-4wrkw\" (UID: \"e390b071-fae1-49c9-879e-be2226eee6a9\") " pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" Jul 27 22:36:06.186905 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:06.186802 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlbkm\" (UniqueName: \"kubernetes.io/projected/e390b071-fae1-49c9-879e-be2226eee6a9-kube-api-access-zlbkm\") pod \"kuadrant-operator-catalog-4wrkw\" (UID: \"e390b071-fae1-49c9-879e-be2226eee6a9\") " pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" Jul 27 22:36:06.196492 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:06.196467 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlbkm\" (UniqueName: \"kubernetes.io/projected/e390b071-fae1-49c9-879e-be2226eee6a9-kube-api-access-zlbkm\") pod \"kuadrant-operator-catalog-4wrkw\" (UID: \"e390b071-fae1-49c9-879e-be2226eee6a9\") " pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" Jul 27 22:36:06.274648 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:06.274614 2561 generic.go:356] "Generic (PLEG): container finished" podID="6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc" containerID="73149f1d33313aeab40c68b8d079e7c709e25c4399d5c12a114449d6243b049e" exitCode=0 Jul 27 22:36:06.274828 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:06.274668 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c" event={"ID":"6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc","Type":"ContainerDied","Data":"73149f1d33313aeab40c68b8d079e7c709e25c4399d5c12a114449d6243b049e"} Jul 27 22:36:06.274828 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:06.274709 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c" event={"ID":"6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc","Type":"ContainerStarted","Data":"a30bf78ddcd8cb85dc09ea47eb7ff2da82850f54a8022bd629d399ecf839103f"} Jul 27 22:36:06.360553 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:06.360516 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" Jul 27 22:36:06.500431 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:06.500406 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4wrkw"] Jul 27 22:36:06.501896 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:36:06.501856 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode390b071_fae1_49c9_879e_be2226eee6a9.slice/crio-a401f0e68c9c2b99ce91cf1d906b9b5844e4f68d0d87b111786e241b2c0af73c WatchSource:0}: Error finding container a401f0e68c9c2b99ce91cf1d906b9b5844e4f68d0d87b111786e241b2c0af73c: Status 404 returned error can't find the container with id a401f0e68c9c2b99ce91cf1d906b9b5844e4f68d0d87b111786e241b2c0af73c Jul 27 22:36:06.503445 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:06.503425 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 22:36:07.281273 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:07.281225 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" event={"ID":"e390b071-fae1-49c9-879e-be2226eee6a9","Type":"ContainerStarted","Data":"1b410c8ab77b437fbeb2c287bffd9620af41ef71745aaae243130750e3396f15"} Jul 27 22:36:07.281760 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:07.281306 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" event={"ID":"e390b071-fae1-49c9-879e-be2226eee6a9","Type":"ContainerStarted","Data":"a401f0e68c9c2b99ce91cf1d906b9b5844e4f68d0d87b111786e241b2c0af73c"} Jul 27 22:36:07.303380 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:07.303318 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" podStartSLOduration=0.91025592 podStartE2EDuration="1.303301666s" totalImagesPullingTime="393.045746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:36:07.30084044 +0000 UTC m=+3044.918277760" watchObservedRunningTime="2026-07-27 22:36:07.303301666 +0000 UTC m=+3044.920738984" Jul 27 22:36:07.414354 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:07.414329 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c" Jul 27 22:36:07.500199 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:07.500169 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc-maas-api-service-ca\") pod \"6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc\" (UID: \"6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc\") " Jul 27 22:36:07.500540 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:07.500518 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc" (UID: "6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:36:07.600928 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:07.600849 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc-maas-api-service-ca\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:36:08.287801 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:08.287772 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c" Jul 27 22:36:08.288209 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:08.287773 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c" event={"ID":"6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc","Type":"ContainerDied","Data":"a30bf78ddcd8cb85dc09ea47eb7ff2da82850f54a8022bd629d399ecf839103f"} Jul 27 22:36:08.288209 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:08.287891 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a30bf78ddcd8cb85dc09ea47eb7ff2da82850f54a8022bd629d399ecf839103f" Jul 27 22:36:15.419157 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:15.419059 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:15.492733 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:15.492698 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:15.554683 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:15.554653 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:16.361573 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:16.361529 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" Jul 27 22:36:16.361738 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:16.361603 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" Jul 27 22:36:16.383375 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:16.383348 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" Jul 27 22:36:17.357770 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:17.357737 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" Jul 27 22:36:17.396241 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:17.396209 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4wrkw"] Jul 27 22:36:19.345939 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:19.345877 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" podUID="e390b071-fae1-49c9-879e-be2226eee6a9" containerName="registry-server" containerID="cri-o://1b410c8ab77b437fbeb2c287bffd9620af41ef71745aaae243130750e3396f15" gracePeriod=2 Jul 27 22:36:19.603574 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:19.603506 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" Jul 27 22:36:19.709808 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:19.709778 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlbkm\" (UniqueName: \"kubernetes.io/projected/e390b071-fae1-49c9-879e-be2226eee6a9-kube-api-access-zlbkm\") pod \"e390b071-fae1-49c9-879e-be2226eee6a9\" (UID: \"e390b071-fae1-49c9-879e-be2226eee6a9\") " Jul 27 22:36:19.711900 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:19.711864 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e390b071-fae1-49c9-879e-be2226eee6a9-kube-api-access-zlbkm" (OuterVolumeSpecName: "kube-api-access-zlbkm") pod "e390b071-fae1-49c9-879e-be2226eee6a9" (UID: "e390b071-fae1-49c9-879e-be2226eee6a9"). InnerVolumeSpecName "kube-api-access-zlbkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:36:19.810732 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:19.810677 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlbkm\" (UniqueName: \"kubernetes.io/projected/e390b071-fae1-49c9-879e-be2226eee6a9-kube-api-access-zlbkm\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:36:19.905967 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:19.905931 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c"] Jul 27 22:36:19.908785 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:19.908759 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:19.909195 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:19.909177 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1368f7-fw99c"] Jul 27 22:36:20.313938 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:20.313847 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:20.353695 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:20.353660 2561 generic.go:356] "Generic (PLEG): container finished" podID="e390b071-fae1-49c9-879e-be2226eee6a9" containerID="1b410c8ab77b437fbeb2c287bffd9620af41ef71745aaae243130750e3396f15" exitCode=0 Jul 27 22:36:20.354148 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:20.353709 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" Jul 27 22:36:20.354148 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:20.353743 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" event={"ID":"e390b071-fae1-49c9-879e-be2226eee6a9","Type":"ContainerDied","Data":"1b410c8ab77b437fbeb2c287bffd9620af41ef71745aaae243130750e3396f15"} Jul 27 22:36:20.354148 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:20.353801 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4wrkw" event={"ID":"e390b071-fae1-49c9-879e-be2226eee6a9","Type":"ContainerDied","Data":"a401f0e68c9c2b99ce91cf1d906b9b5844e4f68d0d87b111786e241b2c0af73c"} Jul 27 22:36:20.354148 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:20.353827 2561 scope.go:117] "RemoveContainer" containerID="1b410c8ab77b437fbeb2c287bffd9620af41ef71745aaae243130750e3396f15" Jul 27 22:36:20.363289 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:20.363252 2561 scope.go:117] "RemoveContainer" containerID="1b410c8ab77b437fbeb2c287bffd9620af41ef71745aaae243130750e3396f15" Jul 27 22:36:20.363542 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:36:20.363524 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b410c8ab77b437fbeb2c287bffd9620af41ef71745aaae243130750e3396f15\": container with ID starting with 1b410c8ab77b437fbeb2c287bffd9620af41ef71745aaae243130750e3396f15 not found: ID does not exist" containerID="1b410c8ab77b437fbeb2c287bffd9620af41ef71745aaae243130750e3396f15" Jul 27 22:36:20.363591 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:20.363551 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b410c8ab77b437fbeb2c287bffd9620af41ef71745aaae243130750e3396f15"} err="failed to get container status \"1b410c8ab77b437fbeb2c287bffd9620af41ef71745aaae243130750e3396f15\": rpc error: code = NotFound desc = could not find container \"1b410c8ab77b437fbeb2c287bffd9620af41ef71745aaae243130750e3396f15\": container with ID starting with 1b410c8ab77b437fbeb2c287bffd9620af41ef71745aaae243130750e3396f15 not found: ID does not exist" Jul 27 22:36:20.375480 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:20.375455 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4wrkw"] Jul 27 22:36:20.379169 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:20.379149 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4wrkw"] Jul 27 22:36:20.818324 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:20.818287 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc" path="/var/lib/kubelet/pods/6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc/volumes" Jul 27 22:36:20.818690 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:20.818674 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e390b071-fae1-49c9-879e-be2226eee6a9" path="/var/lib/kubelet/pods/e390b071-fae1-49c9-879e-be2226eee6a9/volumes" Jul 27 22:36:26.141478 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:26.141444 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:26.242784 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:26.242719 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:26.430586 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:26.430552 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:26.664956 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:26.664914 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:30.328048 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.328010 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6"] Jul 27 22:36:30.328474 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.328377 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e390b071-fae1-49c9-879e-be2226eee6a9" containerName="registry-server" Jul 27 22:36:30.328474 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.328390 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e390b071-fae1-49c9-879e-be2226eee6a9" containerName="registry-server" Jul 27 22:36:30.328474 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.328404 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc" containerName="revoke-keys" Jul 27 22:36:30.328474 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.328410 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc" containerName="revoke-keys" Jul 27 22:36:30.328474 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.328467 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e390b071-fae1-49c9-879e-be2226eee6a9" containerName="registry-server" Jul 27 22:36:30.328649 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.328478 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a2c6cb0-4e23-42b4-a8cb-b4d4ff7034cc" containerName="revoke-keys" Jul 27 22:36:30.332754 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.332736 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6" Jul 27 22:36:30.335146 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.335120 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\"" Jul 27 22:36:30.335323 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.335150 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 22:36:30.339331 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.339303 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6"] Jul 27 22:36:30.409607 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.409569 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b532ae8-04f7-443c-8918-d21eb6d252f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6\" (UID: \"4b532ae8-04f7-443c-8918-d21eb6d252f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6" Jul 27 22:36:30.510208 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.510174 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b532ae8-04f7-443c-8918-d21eb6d252f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6\" (UID: \"4b532ae8-04f7-443c-8918-d21eb6d252f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6" Jul 27 22:36:30.510920 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.510899 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b532ae8-04f7-443c-8918-d21eb6d252f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6\" (UID: \"4b532ae8-04f7-443c-8918-d21eb6d252f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6" Jul 27 22:36:30.646348 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.646247 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6" Jul 27 22:36:30.783132 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:30.783039 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6"] Jul 27 22:36:30.786070 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:36:30.786045 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b532ae8_04f7_443c_8918_d21eb6d252f6.slice/crio-7b4f4c69154a1ebc07bec518d2839d608ef36103ceac23ff8783c8fffbebeb07 WatchSource:0}: Error finding container 7b4f4c69154a1ebc07bec518d2839d608ef36103ceac23ff8783c8fffbebeb07: Status 404 returned error can't find the container with id 7b4f4c69154a1ebc07bec518d2839d608ef36103ceac23ff8783c8fffbebeb07 Jul 27 22:36:31.421644 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:31.421607 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6" event={"ID":"4b532ae8-04f7-443c-8918-d21eb6d252f6","Type":"ContainerStarted","Data":"75707f5d466b28f196fb5ee1db20c2a004ac2040e1e44d76657b8b58285b40f8"} Jul 27 22:36:31.422112 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:31.421667 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6" event={"ID":"4b532ae8-04f7-443c-8918-d21eb6d252f6","Type":"ContainerStarted","Data":"7b4f4c69154a1ebc07bec518d2839d608ef36103ceac23ff8783c8fffbebeb07"} Jul 27 22:36:31.438402 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:31.438311 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6" podStartSLOduration=1.438295379 podStartE2EDuration="1.438295379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:36:31.437152902 +0000 UTC m=+3069.054590221" watchObservedRunningTime="2026-07-27 22:36:31.438295379 +0000 UTC m=+3069.055732697" Jul 27 22:36:32.428161 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:32.428126 2561 generic.go:356] "Generic (PLEG): container finished" podID="4b532ae8-04f7-443c-8918-d21eb6d252f6" containerID="75707f5d466b28f196fb5ee1db20c2a004ac2040e1e44d76657b8b58285b40f8" exitCode=0 Jul 27 22:36:32.428161 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:32.428166 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6" event={"ID":"4b532ae8-04f7-443c-8918-d21eb6d252f6","Type":"ContainerDied","Data":"75707f5d466b28f196fb5ee1db20c2a004ac2040e1e44d76657b8b58285b40f8"} Jul 27 22:36:33.568977 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:33.568951 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6" Jul 27 22:36:33.640139 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:33.640098 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b532ae8-04f7-443c-8918-d21eb6d252f6-maas-api-service-ca\") pod \"4b532ae8-04f7-443c-8918-d21eb6d252f6\" (UID: \"4b532ae8-04f7-443c-8918-d21eb6d252f6\") " Jul 27 22:36:33.640570 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:33.640547 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b532ae8-04f7-443c-8918-d21eb6d252f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4b532ae8-04f7-443c-8918-d21eb6d252f6" (UID: "4b532ae8-04f7-443c-8918-d21eb6d252f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:36:33.741250 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:33.741159 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b532ae8-04f7-443c-8918-d21eb6d252f6-maas-api-service-ca\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:36:34.445873 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:34.445847 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6" Jul 27 22:36:34.446039 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:34.445843 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6" event={"ID":"4b532ae8-04f7-443c-8918-d21eb6d252f6","Type":"ContainerDied","Data":"7b4f4c69154a1ebc07bec518d2839d608ef36103ceac23ff8783c8fffbebeb07"} Jul 27 22:36:34.446039 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:34.445989 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b4f4c69154a1ebc07bec518d2839d608ef36103ceac23ff8783c8fffbebeb07" Jul 27 22:36:45.799589 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:45.799547 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:45.877435 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:45.877397 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:45.939257 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:45.939220 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:50.404279 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:50.404218 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:50.405351 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:50.405327 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6"] Jul 27 22:36:50.407474 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:50.407454 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-626801-w9sf6"] Jul 27 22:36:50.817228 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:50.817127 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:36:50.818766 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:50.818743 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b532ae8-04f7-443c-8918-d21eb6d252f6" path="/var/lib/kubelet/pods/4b532ae8-04f7-443c-8918-d21eb6d252f6/volumes" Jul 27 22:36:55.895495 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:36:55.895460 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:37:29.671462 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:29.671425 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:37:29.727769 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:29.727737 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:37:32.161411 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:32.161363 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:37:33.020140 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:33.020102 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:37:38.618331 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:38.618288 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:37:38.686043 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:38.686009 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:37:38.888089 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:38.888000 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6"] Jul 27 22:37:38.888247 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:38.888222 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerName="istio-proxy" containerID="cri-o://a721523f631b9efc3e459a23e58ad6f9bc070f60e07a797f791e417376546958" gracePeriod=30 Jul 27 22:37:38.935903 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:38.935861 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:37:39.157746 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:39.157710 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:37:40.010317 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:40.010280 2561 patch_prober.go:28] interesting pod/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 22:37:40.010716 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:40.010342 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 22:37:44.143063 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.143031 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:37:44.177511 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.177420 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-workload-socket\") pod \"501b57e9-af8c-423b-81c3-e745fe80e8aa\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " Jul 27 22:37:44.177511 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.177486 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-podinfo\") pod \"501b57e9-af8c-423b-81c3-e745fe80e8aa\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " Jul 27 22:37:44.177738 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.177530 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-data\") pod \"501b57e9-af8c-423b-81c3-e745fe80e8aa\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " Jul 27 22:37:44.177738 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.177566 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501b57e9-af8c-423b-81c3-e745fe80e8aa-istiod-ca-cert\") pod \"501b57e9-af8c-423b-81c3-e745fe80e8aa\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " Jul 27 22:37:44.177738 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.177616 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-workload-certs\") pod \"501b57e9-af8c-423b-81c3-e745fe80e8aa\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " Jul 27 22:37:44.177738 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.177647 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-token\") pod \"501b57e9-af8c-423b-81c3-e745fe80e8aa\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " Jul 27 22:37:44.177738 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.177674 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-credential-socket\") pod \"501b57e9-af8c-423b-81c3-e745fe80e8aa\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " Jul 27 22:37:44.177738 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.177714 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-envoy\") pod \"501b57e9-af8c-423b-81c3-e745fe80e8aa\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " Jul 27 22:37:44.178030 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.177759 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh98d\" (UniqueName: \"kubernetes.io/projected/501b57e9-af8c-423b-81c3-e745fe80e8aa-kube-api-access-vh98d\") pod \"501b57e9-af8c-423b-81c3-e745fe80e8aa\" (UID: \"501b57e9-af8c-423b-81c3-e745fe80e8aa\") " Jul 27 22:37:44.178082 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.178037 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501b57e9-af8c-423b-81c3-e745fe80e8aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "501b57e9-af8c-423b-81c3-e745fe80e8aa" (UID: "501b57e9-af8c-423b-81c3-e745fe80e8aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:37:44.178177 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.178157 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "501b57e9-af8c-423b-81c3-e745fe80e8aa" (UID: "501b57e9-af8c-423b-81c3-e745fe80e8aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:37:44.178296 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.178272 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "501b57e9-af8c-423b-81c3-e745fe80e8aa" (UID: "501b57e9-af8c-423b-81c3-e745fe80e8aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:37:44.178392 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.178328 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "501b57e9-af8c-423b-81c3-e745fe80e8aa" (UID: "501b57e9-af8c-423b-81c3-e745fe80e8aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:37:44.178392 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.178344 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "501b57e9-af8c-423b-81c3-e745fe80e8aa" (UID: "501b57e9-af8c-423b-81c3-e745fe80e8aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:37:44.180148 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.180123 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501b57e9-af8c-423b-81c3-e745fe80e8aa-kube-api-access-vh98d" (OuterVolumeSpecName: "kube-api-access-vh98d") pod "501b57e9-af8c-423b-81c3-e745fe80e8aa" (UID: "501b57e9-af8c-423b-81c3-e745fe80e8aa"). InnerVolumeSpecName "kube-api-access-vh98d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:37:44.180312 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.180246 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "501b57e9-af8c-423b-81c3-e745fe80e8aa" (UID: "501b57e9-af8c-423b-81c3-e745fe80e8aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:37:44.180433 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.180412 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "501b57e9-af8c-423b-81c3-e745fe80e8aa" (UID: "501b57e9-af8c-423b-81c3-e745fe80e8aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:37:44.180508 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.180440 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "501b57e9-af8c-423b-81c3-e745fe80e8aa" (UID: "501b57e9-af8c-423b-81c3-e745fe80e8aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:37:44.279420 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.279370 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:37:44.279420 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.279414 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:37:44.279420 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.279424 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:37:44.279420 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.279435 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh98d\" (UniqueName: \"kubernetes.io/projected/501b57e9-af8c-423b-81c3-e745fe80e8aa-kube-api-access-vh98d\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:37:44.279702 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.279445 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:37:44.279702 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.279453 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:37:44.279702 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.279461 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:37:44.279702 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.279468 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501b57e9-af8c-423b-81c3-e745fe80e8aa-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:37:44.279702 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.279475 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501b57e9-af8c-423b-81c3-e745fe80e8aa-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:37:44.819575 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.819545 2561 generic.go:356] "Generic (PLEG): container finished" podID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerID="a721523f631b9efc3e459a23e58ad6f9bc070f60e07a797f791e417376546958" exitCode=0 Jul 27 22:37:44.819739 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.819587 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" Jul 27 22:37:44.819739 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.819618 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" event={"ID":"501b57e9-af8c-423b-81c3-e745fe80e8aa","Type":"ContainerDied","Data":"a721523f631b9efc3e459a23e58ad6f9bc070f60e07a797f791e417376546958"} Jul 27 22:37:44.819739 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.819678 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6" event={"ID":"501b57e9-af8c-423b-81c3-e745fe80e8aa","Type":"ContainerDied","Data":"561f839f21dac129753272fb8aa5333564ceee7cba6643849f65a69eb654d859"} Jul 27 22:37:44.819739 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.819703 2561 scope.go:117] "RemoveContainer" containerID="a721523f631b9efc3e459a23e58ad6f9bc070f60e07a797f791e417376546958" Jul 27 22:37:44.829956 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.829933 2561 scope.go:117] "RemoveContainer" containerID="c0ce02e60227e3012dae4019244c19369858a06bc68dbbe5dcb8e0bd96fa5862" Jul 27 22:37:44.838484 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.838466 2561 scope.go:117] "RemoveContainer" containerID="a721523f631b9efc3e459a23e58ad6f9bc070f60e07a797f791e417376546958" Jul 27 22:37:44.838713 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:37:44.838697 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a721523f631b9efc3e459a23e58ad6f9bc070f60e07a797f791e417376546958\": container with ID starting with a721523f631b9efc3e459a23e58ad6f9bc070f60e07a797f791e417376546958 not found: ID does not exist" containerID="a721523f631b9efc3e459a23e58ad6f9bc070f60e07a797f791e417376546958" Jul 27 22:37:44.838765 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.838719 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a721523f631b9efc3e459a23e58ad6f9bc070f60e07a797f791e417376546958"} err="failed to get container status \"a721523f631b9efc3e459a23e58ad6f9bc070f60e07a797f791e417376546958\": rpc error: code = NotFound desc = could not find container \"a721523f631b9efc3e459a23e58ad6f9bc070f60e07a797f791e417376546958\": container with ID starting with a721523f631b9efc3e459a23e58ad6f9bc070f60e07a797f791e417376546958 not found: ID does not exist" Jul 27 22:37:44.838765 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.838735 2561 scope.go:117] "RemoveContainer" containerID="c0ce02e60227e3012dae4019244c19369858a06bc68dbbe5dcb8e0bd96fa5862" Jul 27 22:37:44.838946 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:37:44.838927 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ce02e60227e3012dae4019244c19369858a06bc68dbbe5dcb8e0bd96fa5862\": container with ID starting with c0ce02e60227e3012dae4019244c19369858a06bc68dbbe5dcb8e0bd96fa5862 not found: ID does not exist" containerID="c0ce02e60227e3012dae4019244c19369858a06bc68dbbe5dcb8e0bd96fa5862" Jul 27 22:37:44.838987 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.838952 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ce02e60227e3012dae4019244c19369858a06bc68dbbe5dcb8e0bd96fa5862"} err="failed to get container status \"c0ce02e60227e3012dae4019244c19369858a06bc68dbbe5dcb8e0bd96fa5862\": rpc error: code = NotFound desc = could not find container \"c0ce02e60227e3012dae4019244c19369858a06bc68dbbe5dcb8e0bd96fa5862\": container with ID starting with c0ce02e60227e3012dae4019244c19369858a06bc68dbbe5dcb8e0bd96fa5862 not found: ID does not exist" Jul 27 22:37:44.844315 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.844292 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6"] Jul 27 22:37:44.848465 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:44.848443 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-626801-openshift-default-6789f4556-94sp6"] Jul 27 22:37:46.823314 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:46.823254 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" path="/var/lib/kubelet/pods/501b57e9-af8c-423b-81c3-e745fe80e8aa/volumes" Jul 27 22:37:49.799660 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:49.799618 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:37:49.900722 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:49.900687 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:37:49.999693 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:49.999656 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:37:54.367536 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:54.367447 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:37:54.764162 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:37:54.764121 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:00.870697 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:00.870657 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:00.940283 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:00.940223 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:01.149796 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:01.149760 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:01.401662 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:01.401574 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:08.658193 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:08.658148 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:12.009021 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:12.008981 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:12.088699 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:12.088663 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:12.157290 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:12.157221 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:16.626460 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:16.626415 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:17.088315 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:17.088256 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:22.956891 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:22.956850 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:23.060229 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:23.060196 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:23.233903 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:23.233823 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8"] Jul 27 22:38:23.234078 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:23.234055 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" podUID="d43d75b4-e488-448d-a196-50eac2d9a087" containerName="istio-proxy" containerID="cri-o://b98d0a1d3c4602137353e53610e91791553ab5447db531350f96e8a49047ce3c" gracePeriod=30 Jul 27 22:38:23.263334 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:23.263306 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:23.483654 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:23.483618 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:27.979688 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:27.979652 2561 patch_prober.go:28] interesting pod/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 22:38:27.980077 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:27.979713 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" podUID="d43d75b4-e488-448d-a196-50eac2d9a087" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 22:38:28.489050 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.489024 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:38:28.577936 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.577835 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43d75b4-e488-448d-a196-50eac2d9a087-istio-token\") pod \"d43d75b4-e488-448d-a196-50eac2d9a087\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " Jul 27 22:38:28.578118 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.578084 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43d75b4-e488-448d-a196-50eac2d9a087-istio-podinfo\") pod \"d43d75b4-e488-448d-a196-50eac2d9a087\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " Jul 27 22:38:28.578118 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.578115 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-workload-socket\") pod \"d43d75b4-e488-448d-a196-50eac2d9a087\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " Jul 27 22:38:28.578250 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.578159 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-workload-certs\") pod \"d43d75b4-e488-448d-a196-50eac2d9a087\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " Jul 27 22:38:28.578250 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.578179 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-istio-envoy\") pod \"d43d75b4-e488-448d-a196-50eac2d9a087\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " Jul 27 22:38:28.578250 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.578196 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43d75b4-e488-448d-a196-50eac2d9a087-istiod-ca-cert\") pod \"d43d75b4-e488-448d-a196-50eac2d9a087\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " Jul 27 22:38:28.578250 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.578233 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-istio-data\") pod \"d43d75b4-e488-448d-a196-50eac2d9a087\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " Jul 27 22:38:28.578529 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.578299 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpv4v\" (UniqueName: \"kubernetes.io/projected/d43d75b4-e488-448d-a196-50eac2d9a087-kube-api-access-lpv4v\") pod \"d43d75b4-e488-448d-a196-50eac2d9a087\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " Jul 27 22:38:28.578529 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.578491 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-credential-socket\") pod \"d43d75b4-e488-448d-a196-50eac2d9a087\" (UID: \"d43d75b4-e488-448d-a196-50eac2d9a087\") " Jul 27 22:38:28.578640 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.578586 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d43d75b4-e488-448d-a196-50eac2d9a087" (UID: "d43d75b4-e488-448d-a196-50eac2d9a087"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:38:28.578883 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.578773 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d43d75b4-e488-448d-a196-50eac2d9a087" (UID: "d43d75b4-e488-448d-a196-50eac2d9a087"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:38:28.578883 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.578816 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:28.579030 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.579003 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d43d75b4-e488-448d-a196-50eac2d9a087" (UID: "d43d75b4-e488-448d-a196-50eac2d9a087"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:38:28.579135 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.579111 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43d75b4-e488-448d-a196-50eac2d9a087-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d43d75b4-e488-448d-a196-50eac2d9a087" (UID: "d43d75b4-e488-448d-a196-50eac2d9a087"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:38:28.579772 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.579735 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-istio-data" (OuterVolumeSpecName: "istio-data") pod "d43d75b4-e488-448d-a196-50eac2d9a087" (UID: "d43d75b4-e488-448d-a196-50eac2d9a087"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:38:28.580711 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.580685 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43d75b4-e488-448d-a196-50eac2d9a087-istio-token" (OuterVolumeSpecName: "istio-token") pod "d43d75b4-e488-448d-a196-50eac2d9a087" (UID: "d43d75b4-e488-448d-a196-50eac2d9a087"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:38:28.580804 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.580784 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d43d75b4-e488-448d-a196-50eac2d9a087-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d43d75b4-e488-448d-a196-50eac2d9a087" (UID: "d43d75b4-e488-448d-a196-50eac2d9a087"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:38:28.580876 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.580859 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d43d75b4-e488-448d-a196-50eac2d9a087" (UID: "d43d75b4-e488-448d-a196-50eac2d9a087"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:38:28.580936 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.580917 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43d75b4-e488-448d-a196-50eac2d9a087-kube-api-access-lpv4v" (OuterVolumeSpecName: "kube-api-access-lpv4v") pod "d43d75b4-e488-448d-a196-50eac2d9a087" (UID: "d43d75b4-e488-448d-a196-50eac2d9a087"). InnerVolumeSpecName "kube-api-access-lpv4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:38:28.679988 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.679934 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:28.679988 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.679983 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43d75b4-e488-448d-a196-50eac2d9a087-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:28.679988 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.679994 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43d75b4-e488-448d-a196-50eac2d9a087-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:28.679988 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.680003 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:28.679988 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.680010 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:28.680339 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.680019 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43d75b4-e488-448d-a196-50eac2d9a087-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:28.680339 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.680026 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43d75b4-e488-448d-a196-50eac2d9a087-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:28.680339 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:28.680034 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpv4v\" (UniqueName: \"kubernetes.io/projected/d43d75b4-e488-448d-a196-50eac2d9a087-kube-api-access-lpv4v\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:29.050873 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:29.050834 2561 generic.go:356] "Generic (PLEG): container finished" podID="d43d75b4-e488-448d-a196-50eac2d9a087" containerID="b98d0a1d3c4602137353e53610e91791553ab5447db531350f96e8a49047ce3c" exitCode=0 Jul 27 22:38:29.051405 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:29.050908 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" Jul 27 22:38:29.051405 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:29.050922 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" event={"ID":"d43d75b4-e488-448d-a196-50eac2d9a087","Type":"ContainerDied","Data":"b98d0a1d3c4602137353e53610e91791553ab5447db531350f96e8a49047ce3c"} Jul 27 22:38:29.051405 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:29.050996 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8" event={"ID":"d43d75b4-e488-448d-a196-50eac2d9a087","Type":"ContainerDied","Data":"1fd4186bfeddb0805b299495e1581057e715323723f7391f554e1ef6ec34a11d"} Jul 27 22:38:29.051405 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:29.051035 2561 scope.go:117] "RemoveContainer" containerID="b98d0a1d3c4602137353e53610e91791553ab5447db531350f96e8a49047ce3c" Jul 27 22:38:29.060231 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:29.060206 2561 scope.go:117] "RemoveContainer" containerID="0abe9e9f037d6ad1923f74f52d2de41661ceb4082d44b0626b5898b23fb5ea5e" Jul 27 22:38:29.068561 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:29.068542 2561 scope.go:117] "RemoveContainer" containerID="b98d0a1d3c4602137353e53610e91791553ab5447db531350f96e8a49047ce3c" Jul 27 22:38:29.068870 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:38:29.068849 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98d0a1d3c4602137353e53610e91791553ab5447db531350f96e8a49047ce3c\": container with ID starting with b98d0a1d3c4602137353e53610e91791553ab5447db531350f96e8a49047ce3c not found: ID does not exist" containerID="b98d0a1d3c4602137353e53610e91791553ab5447db531350f96e8a49047ce3c" Jul 27 22:38:29.068933 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:29.068879 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98d0a1d3c4602137353e53610e91791553ab5447db531350f96e8a49047ce3c"} err="failed to get container status \"b98d0a1d3c4602137353e53610e91791553ab5447db531350f96e8a49047ce3c\": rpc error: code = NotFound desc = could not find container \"b98d0a1d3c4602137353e53610e91791553ab5447db531350f96e8a49047ce3c\": container with ID starting with b98d0a1d3c4602137353e53610e91791553ab5447db531350f96e8a49047ce3c not found: ID does not exist" Jul 27 22:38:29.068933 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:29.068898 2561 scope.go:117] "RemoveContainer" containerID="0abe9e9f037d6ad1923f74f52d2de41661ceb4082d44b0626b5898b23fb5ea5e" Jul 27 22:38:29.069159 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:38:29.069141 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0abe9e9f037d6ad1923f74f52d2de41661ceb4082d44b0626b5898b23fb5ea5e\": container with ID starting with 0abe9e9f037d6ad1923f74f52d2de41661ceb4082d44b0626b5898b23fb5ea5e not found: ID does not exist" containerID="0abe9e9f037d6ad1923f74f52d2de41661ceb4082d44b0626b5898b23fb5ea5e" Jul 27 22:38:29.069205 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:29.069165 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0abe9e9f037d6ad1923f74f52d2de41661ceb4082d44b0626b5898b23fb5ea5e"} err="failed to get container status \"0abe9e9f037d6ad1923f74f52d2de41661ceb4082d44b0626b5898b23fb5ea5e\": rpc error: code = NotFound desc = could not find container \"0abe9e9f037d6ad1923f74f52d2de41661ceb4082d44b0626b5898b23fb5ea5e\": container with ID starting with 0abe9e9f037d6ad1923f74f52d2de41661ceb4082d44b0626b5898b23fb5ea5e not found: ID does not exist" Jul 27 22:38:29.069630 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:29.069612 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8"] Jul 27 22:38:29.073532 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:29.073510 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-6138cc-openshift-default-79d4489876-8lrv8"] Jul 27 22:38:30.818199 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:30.818151 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d43d75b4-e488-448d-a196-50eac2d9a087" path="/var/lib/kubelet/pods/d43d75b4-e488-448d-a196-50eac2d9a087/volumes" Jul 27 22:38:34.124570 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:34.124539 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:34.267951 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:34.267911 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:34.494816 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:34.494771 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:38.696746 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:38.696717 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:39.899504 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:39.899469 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:45.105080 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:45.105038 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:45.203226 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:45.203184 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:45.369309 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:45.369204 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp"] Jul 27 22:38:45.369510 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:45.369484 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" containerName="istio-proxy" containerID="cri-o://ef03af865fc4460331a804257a68dce2bc4979c151227686635de88626fab666" gracePeriod=30 Jul 27 22:38:45.387726 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:45.387686 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:45.627360 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:45.627257 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:46.083522 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:46.083491 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-796c9964dc-dxhvw_d3aec408-771b-409f-b46d-02197bf53a7c/authorino/0.log" Jul 27 22:38:49.065251 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:49.065219 2561 patch_prober.go:28] interesting pod/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 22:38:49.065706 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:49.065296 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 22:38:50.619345 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.619322 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:38:50.677406 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.677375 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-istio-envoy\") pod \"9b18b192-a462-4649-9416-c411c15ad5f2\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " Jul 27 22:38:50.677605 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.677420 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b18b192-a462-4649-9416-c411c15ad5f2-istio-token\") pod \"9b18b192-a462-4649-9416-c411c15ad5f2\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " Jul 27 22:38:50.677605 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.677450 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6w25\" (UniqueName: \"kubernetes.io/projected/9b18b192-a462-4649-9416-c411c15ad5f2-kube-api-access-c6w25\") pod \"9b18b192-a462-4649-9416-c411c15ad5f2\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " Jul 27 22:38:50.677605 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.677544 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-workload-socket\") pod \"9b18b192-a462-4649-9416-c411c15ad5f2\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " Jul 27 22:38:50.677605 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.677579 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-istio-data\") pod \"9b18b192-a462-4649-9416-c411c15ad5f2\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " Jul 27 22:38:50.677605 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.677601 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b18b192-a462-4649-9416-c411c15ad5f2-istio-podinfo\") pod \"9b18b192-a462-4649-9416-c411c15ad5f2\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " Jul 27 22:38:50.677904 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.677621 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-workload-certs\") pod \"9b18b192-a462-4649-9416-c411c15ad5f2\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " Jul 27 22:38:50.677904 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.677642 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-credential-socket\") pod \"9b18b192-a462-4649-9416-c411c15ad5f2\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " Jul 27 22:38:50.677904 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.677822 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b18b192-a462-4649-9416-c411c15ad5f2" (UID: "9b18b192-a462-4649-9416-c411c15ad5f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:38:50.677904 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.677863 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b18b192-a462-4649-9416-c411c15ad5f2-istiod-ca-cert\") pod \"9b18b192-a462-4649-9416-c411c15ad5f2\" (UID: \"9b18b192-a462-4649-9416-c411c15ad5f2\") " Jul 27 22:38:50.678110 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.678032 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b18b192-a462-4649-9416-c411c15ad5f2" (UID: "9b18b192-a462-4649-9416-c411c15ad5f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:38:50.678167 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.678121 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b18b192-a462-4649-9416-c411c15ad5f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b18b192-a462-4649-9416-c411c15ad5f2" (UID: "9b18b192-a462-4649-9416-c411c15ad5f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:38:50.678303 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.678244 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b18b192-a462-4649-9416-c411c15ad5f2" (UID: "9b18b192-a462-4649-9416-c411c15ad5f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:38:50.678445 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.678359 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:50.678445 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.678380 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b18b192-a462-4649-9416-c411c15ad5f2-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:50.678445 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.678393 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:50.678445 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.678401 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b18b192-a462-4649-9416-c411c15ad5f2" (UID: "9b18b192-a462-4649-9416-c411c15ad5f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:38:50.679737 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.679712 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b18b192-a462-4649-9416-c411c15ad5f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b18b192-a462-4649-9416-c411c15ad5f2" (UID: "9b18b192-a462-4649-9416-c411c15ad5f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:38:50.679830 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.679736 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b18b192-a462-4649-9416-c411c15ad5f2" (UID: "9b18b192-a462-4649-9416-c411c15ad5f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:38:50.679830 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.679743 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b18b192-a462-4649-9416-c411c15ad5f2-kube-api-access-c6w25" (OuterVolumeSpecName: "kube-api-access-c6w25") pod "9b18b192-a462-4649-9416-c411c15ad5f2" (UID: "9b18b192-a462-4649-9416-c411c15ad5f2"). InnerVolumeSpecName "kube-api-access-c6w25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:38:50.679902 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.679886 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b18b192-a462-4649-9416-c411c15ad5f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b18b192-a462-4649-9416-c411c15ad5f2" (UID: "9b18b192-a462-4649-9416-c411c15ad5f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:38:50.779178 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.779134 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:50.779178 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.779168 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b18b192-a462-4649-9416-c411c15ad5f2-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:50.779178 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.779178 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:50.779178 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.779187 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b18b192-a462-4649-9416-c411c15ad5f2-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:50.779526 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.779197 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b18b192-a462-4649-9416-c411c15ad5f2-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:50.779526 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:50.779211 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6w25\" (UniqueName: \"kubernetes.io/projected/9b18b192-a462-4649-9416-c411c15ad5f2-kube-api-access-c6w25\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 27 22:38:51.167093 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:51.167060 2561 generic.go:356] "Generic (PLEG): container finished" podID="9b18b192-a462-4649-9416-c411c15ad5f2" containerID="ef03af865fc4460331a804257a68dce2bc4979c151227686635de88626fab666" exitCode=0 Jul 27 22:38:51.167288 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:51.167116 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" Jul 27 22:38:51.167288 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:51.167145 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" event={"ID":"9b18b192-a462-4649-9416-c411c15ad5f2","Type":"ContainerDied","Data":"ef03af865fc4460331a804257a68dce2bc4979c151227686635de88626fab666"} Jul 27 22:38:51.167288 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:51.167194 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp" event={"ID":"9b18b192-a462-4649-9416-c411c15ad5f2","Type":"ContainerDied","Data":"e8f5707a1c6d1b7ba064ce12a84bba58140e9f869f3d14b9722a419785c32eca"} Jul 27 22:38:51.167288 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:51.167216 2561 scope.go:117] "RemoveContainer" containerID="ef03af865fc4460331a804257a68dce2bc4979c151227686635de88626fab666" Jul 27 22:38:51.178624 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:51.178562 2561 scope.go:117] "RemoveContainer" containerID="2bad1e735e4361b48e692746d4c6b0f7fb6b4814b0c4d8a9ee4fa4d1fef45d70" Jul 27 22:38:51.184542 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:51.184514 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp"] Jul 27 22:38:51.187644 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:51.187621 2561 scope.go:117] "RemoveContainer" containerID="ef03af865fc4460331a804257a68dce2bc4979c151227686635de88626fab666" Jul 27 22:38:51.187930 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:38:51.187908 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef03af865fc4460331a804257a68dce2bc4979c151227686635de88626fab666\": container with ID starting with ef03af865fc4460331a804257a68dce2bc4979c151227686635de88626fab666 not found: ID does not exist" containerID="ef03af865fc4460331a804257a68dce2bc4979c151227686635de88626fab666" Jul 27 22:38:51.188015 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:51.187939 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef03af865fc4460331a804257a68dce2bc4979c151227686635de88626fab666"} err="failed to get container status \"ef03af865fc4460331a804257a68dce2bc4979c151227686635de88626fab666\": rpc error: code = NotFound desc = could not find container \"ef03af865fc4460331a804257a68dce2bc4979c151227686635de88626fab666\": container with ID starting with ef03af865fc4460331a804257a68dce2bc4979c151227686635de88626fab666 not found: ID does not exist" Jul 27 22:38:51.188015 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:51.187957 2561 scope.go:117] "RemoveContainer" containerID="2bad1e735e4361b48e692746d4c6b0f7fb6b4814b0c4d8a9ee4fa4d1fef45d70" Jul 27 22:38:51.188218 ip-10-0-129-80 kubenswrapper[2561]: E0727 22:38:51.188201 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bad1e735e4361b48e692746d4c6b0f7fb6b4814b0c4d8a9ee4fa4d1fef45d70\": container with ID starting with 2bad1e735e4361b48e692746d4c6b0f7fb6b4814b0c4d8a9ee4fa4d1fef45d70 not found: ID does not exist" containerID="2bad1e735e4361b48e692746d4c6b0f7fb6b4814b0c4d8a9ee4fa4d1fef45d70" Jul 27 22:38:51.188290 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:51.188224 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bad1e735e4361b48e692746d4c6b0f7fb6b4814b0c4d8a9ee4fa4d1fef45d70"} err="failed to get container status \"2bad1e735e4361b48e692746d4c6b0f7fb6b4814b0c4d8a9ee4fa4d1fef45d70\": rpc error: code = NotFound desc = could not find container \"2bad1e735e4361b48e692746d4c6b0f7fb6b4814b0c4d8a9ee4fa4d1fef45d70\": container with ID starting with 2bad1e735e4361b48e692746d4c6b0f7fb6b4814b0c4d8a9ee4fa4d1fef45d70 not found: ID does not exist" Jul 27 22:38:51.189689 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:51.189671 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-0c79f1-openshift-default-77f467fd8d-9k2mp"] Jul 27 22:38:52.662045 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:52.662016 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n_52d414f2-e4c1-4f85-a815-58fc04fbd99c/copy-manifests/0.log" Jul 27 22:38:52.670290 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:52.670248 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n_52d414f2-e4c1-4f85-a815-58fc04fbd99c/manager/0.log" Jul 27 22:38:52.819705 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:52.819671 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" path="/var/lib/kubelet/pods/9b18b192-a462-4649-9416-c411c15ad5f2/volumes" Jul 27 22:38:53.388033 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:53.388006 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753175-57cgb_b6e13204-85bb-47fd-bc7e-7cebb369ab1e/cleanup/0.log" Jul 27 22:38:54.481656 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:54.481623 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6_16d3de7a-afc6-46bd-8bbf-2833180ba288/util/0.log" Jul 27 22:38:54.489246 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:54.489210 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6_16d3de7a-afc6-46bd-8bbf-2833180ba288/pull/0.log" Jul 27 22:38:54.496107 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:54.496089 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6_16d3de7a-afc6-46bd-8bbf-2833180ba288/extract/0.log" Jul 27 22:38:54.622522 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:54.622492 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk_71695e96-7c86-47b9-a806-cfed10d81d7d/util/0.log" Jul 27 22:38:54.630118 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:54.630092 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk_71695e96-7c86-47b9-a806-cfed10d81d7d/pull/0.log" Jul 27 22:38:54.637496 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:54.637473 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk_71695e96-7c86-47b9-a806-cfed10d81d7d/extract/0.log" Jul 27 22:38:54.758368 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:54.758290 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf_2df13706-8850-4edb-b6b9-9495738b4b95/util/0.log" Jul 27 22:38:54.765468 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:54.765440 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf_2df13706-8850-4edb-b6b9-9495738b4b95/pull/0.log" Jul 27 22:38:54.772717 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:54.772695 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf_2df13706-8850-4edb-b6b9-9495738b4b95/extract/0.log" Jul 27 22:38:54.901393 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:54.901363 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx_bf5626a4-3553-47be-b521-41df7e2bbd9d/util/0.log" Jul 27 22:38:54.908879 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:54.908855 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx_bf5626a4-3553-47be-b521-41df7e2bbd9d/pull/0.log" Jul 27 22:38:54.915926 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:54.915902 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx_bf5626a4-3553-47be-b521-41df7e2bbd9d/extract/0.log" Jul 27 22:38:55.050422 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:55.050336 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-796c9964dc-dxhvw_d3aec408-771b-409f-b46d-02197bf53a7c/authorino/0.log" Jul 27 22:38:55.180839 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:55.180808 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-6kwj2_2af4f0ad-089c-42d9-9a9d-5f8225eddb3b/manager/0.log" Jul 27 22:38:55.428795 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:55.428766 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-hb6h4_de01ca40-aac1-44a2-b1a9-e8bafd04dd46/kuadrant-console-plugin/0.log" Jul 27 22:38:55.561159 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:55.561129 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-k9xnl_57c2f4c0-758c-4ff9-9e75-2437780c3f62/registry-server/0.log" Jul 27 22:38:55.951341 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:55.951309 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-9jvjd_453577e3-d603-4333-b7eb-53d08b661966/manager/0.log" Jul 27 22:38:56.327739 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:56.327650 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cszb2b_b1b40f45-4e8e-4638-8858-c9c8114baeef/istio-proxy/0.log" Jul 27 22:38:56.849094 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:56.849062 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9s7wb_d7f52e88-52c0-4a77-b1ff-bbb40045d05d/istio-proxy/1.log" Jul 27 22:38:57.213606 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:57.213575 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b55db7595-rmmgf_77c6bb84-f632-4621-958c-51b870956328/router/0.log" Jul 27 22:38:57.703714 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:57.703676 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-lhrgh_a4226680-c358-4312-b3fe-142a709a11bb/main/0.log" Jul 27 22:38:58.079500 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:38:58.079423 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-d2hmd_7778340f-8115-4158-871d-19cfc605562c/main/0.log" Jul 27 22:39:03.905096 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:03.905057 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:39:03.998684 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:03.998644 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" limit=1024 Jul 27 22:39:06.803006 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:06.802972 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vlh9m_1ef19975-ecaa-49ba-9cd9-e16737e46c47/global-pull-secret-syncer/0.log" Jul 27 22:39:06.865120 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:06.865083 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6hjdx_6aaee722-f9e3-4563-addf-2a4fe9f5f0a0/konnectivity-agent/0.log" Jul 27 22:39:06.983041 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:06.983014 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-80.ec2.internal_f186ee96a5b10264dea34206a4faa43e/haproxy/0.log" Jul 27 22:39:11.555625 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:11.555535 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6_16d3de7a-afc6-46bd-8bbf-2833180ba288/extract/0.log" Jul 27 22:39:11.581909 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:11.581874 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6_16d3de7a-afc6-46bd-8bbf-2833180ba288/util/0.log" Jul 27 22:39:11.605582 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:11.605547 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75967rt6_16d3de7a-afc6-46bd-8bbf-2833180ba288/pull/0.log" Jul 27 22:39:11.634731 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:11.634704 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk_71695e96-7c86-47b9-a806-cfed10d81d7d/extract/0.log" Jul 27 22:39:11.657733 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:11.657706 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk_71695e96-7c86-47b9-a806-cfed10d81d7d/util/0.log" Jul 27 22:39:11.681214 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:11.681185 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t5tlk_71695e96-7c86-47b9-a806-cfed10d81d7d/pull/0.log" Jul 27 22:39:11.715403 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:11.715371 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf_2df13706-8850-4edb-b6b9-9495738b4b95/extract/0.log" Jul 27 22:39:11.740530 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:11.740501 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf_2df13706-8850-4edb-b6b9-9495738b4b95/util/0.log" Jul 27 22:39:11.765363 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:11.765337 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737vbrf_2df13706-8850-4edb-b6b9-9495738b4b95/pull/0.log" Jul 27 22:39:11.794930 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:11.794898 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx_bf5626a4-3553-47be-b521-41df7e2bbd9d/extract/0.log" Jul 27 22:39:11.824218 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:11.824142 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx_bf5626a4-3553-47be-b521-41df7e2bbd9d/util/0.log" Jul 27 22:39:11.850254 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:11.850228 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcvsx_bf5626a4-3553-47be-b521-41df7e2bbd9d/pull/0.log" Jul 27 22:39:12.446742 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:12.446708 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-796c9964dc-dxhvw_d3aec408-771b-409f-b46d-02197bf53a7c/authorino/0.log" Jul 27 22:39:12.532715 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:12.532688 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-6kwj2_2af4f0ad-089c-42d9-9a9d-5f8225eddb3b/manager/0.log" Jul 27 22:39:12.593198 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:12.593176 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-hb6h4_de01ca40-aac1-44a2-b1a9-e8bafd04dd46/kuadrant-console-plugin/0.log" Jul 27 22:39:12.678741 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:12.678708 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-k9xnl_57c2f4c0-758c-4ff9-9e75-2437780c3f62/registry-server/0.log" Jul 27 22:39:13.041014 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:13.040977 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-9jvjd_453577e3-d603-4333-b7eb-53d08b661966/manager/0.log" Jul 27 22:39:15.035914 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:15.035878 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-l9zg5_8bc9020b-6a51-4caa-9163-0d474d41b484/cluster-monitoring-operator/0.log" Jul 27 22:39:15.318672 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:15.318567 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pqblm_07edee64-5950-439c-9488-6c0c7a5f0191/node-exporter/0.log" Jul 27 22:39:15.341166 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:15.341136 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pqblm_07edee64-5950-439c-9488-6c0c7a5f0191/kube-rbac-proxy/0.log" Jul 27 22:39:15.364340 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:15.364316 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pqblm_07edee64-5950-439c-9488-6c0c7a5f0191/init-textfile/0.log" Jul 27 22:39:17.333343 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.333307 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s"] Jul 27 22:39:17.333842 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.333788 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d43d75b4-e488-448d-a196-50eac2d9a087" containerName="istio-proxy" Jul 27 22:39:17.333842 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.333806 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d43d75b4-e488-448d-a196-50eac2d9a087" containerName="istio-proxy" Jul 27 22:39:17.333842 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.333824 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerName="istio-proxy" Jul 27 22:39:17.333842 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.333832 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerName="istio-proxy" Jul 27 22:39:17.333842 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.333844 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b532ae8-04f7-443c-8918-d21eb6d252f6" containerName="revoke-keys" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.333853 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b532ae8-04f7-443c-8918-d21eb6d252f6" containerName="revoke-keys" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.333867 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerName="istio-proxy" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.333875 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerName="istio-proxy" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.333895 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" containerName="istio-proxy" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.333903 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" containerName="istio-proxy" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.333917 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" containerName="istio-proxy" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.333925 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" containerName="istio-proxy" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.334007 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" containerName="istio-proxy" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.334019 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d43d75b4-e488-448d-a196-50eac2d9a087" containerName="istio-proxy" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.334028 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d43d75b4-e488-448d-a196-50eac2d9a087" containerName="istio-proxy" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.334038 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerName="istio-proxy" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.334049 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b18b192-a462-4649-9416-c411c15ad5f2" containerName="istio-proxy" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.334082 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="501b57e9-af8c-423b-81c3-e745fe80e8aa" containerName="istio-proxy" Jul 27 22:39:17.334128 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.334095 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b532ae8-04f7-443c-8918-d21eb6d252f6" containerName="revoke-keys" Jul 27 22:39:17.337384 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.337365 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.341022 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.341002 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dhm4t\"/\"default-dockercfg-v997v\"" Jul 27 22:39:17.351610 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.351583 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhm4t\"/\"kube-root-ca.crt\"" Jul 27 22:39:17.351977 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.351951 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhm4t\"/\"openshift-service-ca.crt\"" Jul 27 22:39:17.354068 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.354049 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s"] Jul 27 22:39:17.410428 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.410392 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1161ed17-1dd5-4393-a195-6de3d9746df5-proc\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.410428 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.410438 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drscl\" (UniqueName: \"kubernetes.io/projected/1161ed17-1dd5-4393-a195-6de3d9746df5-kube-api-access-drscl\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.410671 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.410482 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1161ed17-1dd5-4393-a195-6de3d9746df5-lib-modules\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.410671 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.410517 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1161ed17-1dd5-4393-a195-6de3d9746df5-sys\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.410671 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.410546 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1161ed17-1dd5-4393-a195-6de3d9746df5-podres\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.511751 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.511716 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drscl\" (UniqueName: \"kubernetes.io/projected/1161ed17-1dd5-4393-a195-6de3d9746df5-kube-api-access-drscl\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.511912 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.511778 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1161ed17-1dd5-4393-a195-6de3d9746df5-lib-modules\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.511912 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.511811 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1161ed17-1dd5-4393-a195-6de3d9746df5-sys\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.511912 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.511842 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1161ed17-1dd5-4393-a195-6de3d9746df5-podres\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.511912 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.511863 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1161ed17-1dd5-4393-a195-6de3d9746df5-proc\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.512134 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.511925 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1161ed17-1dd5-4393-a195-6de3d9746df5-proc\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.512134 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.511934 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1161ed17-1dd5-4393-a195-6de3d9746df5-sys\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.512134 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.511989 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1161ed17-1dd5-4393-a195-6de3d9746df5-lib-modules\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.512134 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.511998 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1161ed17-1dd5-4393-a195-6de3d9746df5-podres\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.520872 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.520844 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drscl\" (UniqueName: \"kubernetes.io/projected/1161ed17-1dd5-4393-a195-6de3d9746df5-kube-api-access-drscl\") pod \"perf-node-gather-daemonset-bwg9s\" (UID: \"1161ed17-1dd5-4393-a195-6de3d9746df5\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.647250 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.647158 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:17.785120 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:17.785087 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s"] Jul 27 22:39:17.788774 ip-10-0-129-80 kubenswrapper[2561]: W0727 22:39:17.788745 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1161ed17_1dd5_4393_a195_6de3d9746df5.slice/crio-99a41e60b588d1d34dd7eb1b14a5f946efdcb9fc434e2ff35861eb4d3879175b WatchSource:0}: Error finding container 99a41e60b588d1d34dd7eb1b14a5f946efdcb9fc434e2ff35861eb4d3879175b: Status 404 returned error can't find the container with id 99a41e60b588d1d34dd7eb1b14a5f946efdcb9fc434e2ff35861eb4d3879175b Jul 27 22:39:18.310680 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:18.310584 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" event={"ID":"1161ed17-1dd5-4393-a195-6de3d9746df5","Type":"ContainerStarted","Data":"25125c5b5c345389e814489d0330c451c943fcc8e90299df71a0817437da2ec8"} Jul 27 22:39:18.310680 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:18.310638 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" event={"ID":"1161ed17-1dd5-4393-a195-6de3d9746df5","Type":"ContainerStarted","Data":"99a41e60b588d1d34dd7eb1b14a5f946efdcb9fc434e2ff35861eb4d3879175b"} Jul 27 22:39:18.310899 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:18.310683 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:18.327946 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:18.327903 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" podStartSLOduration=1.327888883 podStartE2EDuration="1.327888883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 22:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:39:18.327631666 +0000 UTC m=+3235.945068977" watchObservedRunningTime="2026-07-27 22:39:18.327888883 +0000 UTC m=+3235.945326225" Jul 27 22:39:20.468067 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:20.468038 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c77xc_3c159765-958d-4936-a11a-092a2566089b/dns/0.log" Jul 27 22:39:20.490236 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:20.490206 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c77xc_3c159765-958d-4936-a11a-092a2566089b/kube-rbac-proxy/0.log" Jul 27 22:39:20.564319 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:20.564286 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8m8z2_7f949673-96be-4e7f-9cf8-5f69124a6309/dns-node-resolver/0.log" Jul 27 22:39:21.200541 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:21.200508 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-774b54654c-9kg49_157ac3ef-1c87-44e5-8c59-edd637bc0c48/registry/0.log" Jul 27 22:39:21.254130 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:21.254100 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ghrhz_8aa2c150-1b16-43c8-81a7-a290299036d3/node-ca/0.log" Jul 27 22:39:22.260018 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:22.259981 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cszb2b_b1b40f45-4e8e-4638-8858-c9c8114baeef/istio-proxy/0.log" Jul 27 22:39:22.619059 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:22.618966 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9s7wb_d7f52e88-52c0-4a77-b1ff-bbb40045d05d/istio-proxy/0.log" Jul 27 22:39:22.670939 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:22.670906 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9s7wb_d7f52e88-52c0-4a77-b1ff-bbb40045d05d/istio-proxy/1.log" Jul 27 22:39:22.783599 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:22.783572 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b55db7595-rmmgf_77c6bb84-f632-4621-958c-51b870956328/router/0.log" Jul 27 22:39:23.683797 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:23.683764 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z2qwk_8529dd01-316a-4c4b-af14-e04511a99734/serve-healthcheck-canary/0.log" Jul 27 22:39:24.199196 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:24.199169 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-b8v9l_d3596aa9-4bcf-4506-aff0-3706719f73c1/insights-operator/0.log" Jul 27 22:39:24.309746 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:24.309717 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w78fm_a416c09a-03d4-4e4f-995d-217588936dde/kube-rbac-proxy/0.log" Jul 27 22:39:24.324744 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:24.324722 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-bwg9s" Jul 27 22:39:24.339403 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:24.339377 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w78fm_a416c09a-03d4-4e4f-995d-217588936dde/exporter/0.log" Jul 27 22:39:24.366994 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:24.366971 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w78fm_a416c09a-03d4-4e4f-995d-217588936dde/extractor/0.log" Jul 27 22:39:27.823399 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:27.823289 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n_52d414f2-e4c1-4f85-a815-58fc04fbd99c/manager/0.log" Jul 27 22:39:27.866248 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:27.866221 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6d4bf4d4-tt28n_52d414f2-e4c1-4f85-a815-58fc04fbd99c/copy-manifests/0.log" Jul 27 22:39:28.329518 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:28.329488 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cszb2b_b1b40f45-4e8e-4638-8858-c9c8114baeef/istio-proxy/0.log" Jul 27 22:39:28.680880 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:28.680850 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9s7wb_d7f52e88-52c0-4a77-b1ff-bbb40045d05d/istio-proxy/0.log" Jul 27 22:39:28.733831 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:28.733793 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9s7wb_d7f52e88-52c0-4a77-b1ff-bbb40045d05d/istio-proxy/1.log" Jul 27 22:39:28.818523 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:28.818492 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b55db7595-rmmgf_77c6bb84-f632-4621-958c-51b870956328/router/0.log" Jul 27 22:39:29.436547 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:29.436515 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-647fb87ffb-7tdxs_e73b8505-9c44-40f5-a50a-297a426ebb3e/manager/1.log" Jul 27 22:39:29.440923 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:29.440902 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-647fb87ffb-7tdxs_e73b8505-9c44-40f5-a50a-297a426ebb3e/manager/0.log" Jul 27 22:39:30.246313 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:30.246281 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cszb2b_b1b40f45-4e8e-4638-8858-c9c8114baeef/istio-proxy/0.log" Jul 27 22:39:30.570508 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:30.570417 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9s7wb_d7f52e88-52c0-4a77-b1ff-bbb40045d05d/istio-proxy/0.log" Jul 27 22:39:30.622461 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:30.622425 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9s7wb_d7f52e88-52c0-4a77-b1ff-bbb40045d05d/istio-proxy/1.log" Jul 27 22:39:30.708382 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:30.708355 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b55db7595-rmmgf_77c6bb84-f632-4621-958c-51b870956328/router/0.log" Jul 27 22:39:34.693132 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:34.693091 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-82g59_f03b7af1-4768-402b-8b41-f2c48e1a51db/migrator/0.log" Jul 27 22:39:34.717133 ip-10-0-129-80 kubenswrapper[2561]: I0727 22:39:34.717107 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-82g59_f03b7af1-4768-402b-8b41-f2c48e1a51db/graceful-termination/0.log"