Jul 10 22:22:16.054519 ip-10-0-136-68 systemd[1]: Starting Kubernetes Kubelet... Jul 10 22:22:17.458751 ip-10-0-136-68 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 22:22:17.458751 ip-10-0-136-68 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 22:22:17.458751 ip-10-0-136-68 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 22:22:17.458751 ip-10-0-136-68 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 22:22:17.458751 ip-10-0-136-68 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 22:22:17.459756 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.459579 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 22:22:17.463467 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463441 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:22:17.463467 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463461 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:22:17.463467 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463467 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:22:17.463467 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463472 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:22:17.463467 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463477 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463482 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463486 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463492 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463498 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463503 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463507 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463518 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463524 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463530 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463537 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463542 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463547 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463552 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463556 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463560 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463565 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463569 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463574 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:22:17.463775 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463578 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463583 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463587 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463591 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463596 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463601 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463605 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463610 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463614 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463618 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463621 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463626 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463630 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463635 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463639 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463645 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463650 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463654 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463659 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463664 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:22:17.464599 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463669 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463674 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463678 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463683 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463687 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463692 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463696 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463702 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463706 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463711 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463715 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463720 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463724 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463729 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463734 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463739 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463743 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463748 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463752 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:22:17.465335 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463757 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463761 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463766 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463772 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463776 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463780 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463784 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463789 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463794 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463800 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463806 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463811 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463815 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463820 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463824 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463829 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463833 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463838 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463842 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463847 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:22:17.465831 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463852 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463856 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463861 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463882 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463888 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463893 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463897 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463902 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463907 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463911 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463916 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463921 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463925 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463930 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463935 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463939 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.463944 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465253 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465269 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:22:17.466411 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465273 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465277 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465279 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465283 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465286 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465290 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465293 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465295 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465298 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465301 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465304 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465307 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465312 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465317 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465320 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465323 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465326 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465329 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465332 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465335 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:22:17.466957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465338 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465341 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465344 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465347 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465351 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465354 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465357 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465360 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465362 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465365 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465368 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465373 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465376 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465379 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465384 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465388 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465392 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465395 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465398 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:22:17.467513 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465401 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465404 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465406 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465409 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465413 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465416 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465419 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465424 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465427 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465431 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465434 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465437 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465441 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465443 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465446 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465449 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465452 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465454 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465457 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:22:17.468037 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465460 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465463 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465466 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465468 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465471 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465475 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465479 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465482 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465487 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465490 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465493 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465495 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465498 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465501 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465504 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465506 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465509 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465512 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465515 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465517 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:22:17.468603 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465520 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465523 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465525 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465528 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465531 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465534 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465536 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465539 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465541 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465544 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465547 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465550 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465552 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465555 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465558 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465561 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465564 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465568 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.465571 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466011 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:22:17.469140 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466017 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466020 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466024 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466027 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466030 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466033 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466036 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466039 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466044 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466048 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466051 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466054 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466057 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466060 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466063 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466066 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466069 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466071 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466074 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466077 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:22:17.469652 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466080 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466083 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466086 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466088 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466091 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466094 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466097 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466100 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466102 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466107 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466110 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466114 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466116 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466121 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466125 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466128 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466131 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466134 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466137 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:22:17.470189 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466140 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466143 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466146 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466149 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466152 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466155 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466158 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466161 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466164 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466167 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466170 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466172 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466175 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466178 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466181 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466183 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466187 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466189 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466192 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466195 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:22:17.470667 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466198 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466201 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466205 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466208 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466210 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466213 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466216 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466219 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466222 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466224 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466227 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466231 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466235 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466238 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466241 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466243 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466246 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466249 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466252 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466254 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:22:17.471205 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466257 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466260 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466263 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466265 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466268 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466271 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466274 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466277 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466279 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466282 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466285 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466287 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466290 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466293 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466296 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466299 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466302 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466304 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.466307 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466388 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 22:22:17.471721 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466395 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466415 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466428 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466433 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466436 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466441 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466446 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466449 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466453 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466457 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466461 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466464 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466467 2565 flags.go:64] FLAG: --cgroup-root="" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466471 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466474 2565 flags.go:64] FLAG: --client-ca-file="" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466477 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466481 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466485 2565 flags.go:64] FLAG: --cluster-domain="" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466489 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466492 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466495 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466499 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466503 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 22:22:17.472257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466506 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466509 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466516 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466519 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466522 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466525 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466529 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466533 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466538 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466542 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466545 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466548 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466551 2565 flags.go:64] FLAG: --enable-server="true" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466555 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466560 2565 flags.go:64] FLAG: --event-burst="100" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466563 2565 flags.go:64] FLAG: --event-qps="50" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466567 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466570 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466576 2565 flags.go:64] FLAG: --eviction-hard="" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466581 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466584 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466587 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466590 2565 flags.go:64] FLAG: --eviction-soft="" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466594 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466597 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 22:22:17.472858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466600 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466604 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466607 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466610 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466613 2565 flags.go:64] FLAG: --feature-gates="" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466617 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466620 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466624 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466627 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466632 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466635 2565 flags.go:64] FLAG: --help="false" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466638 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466641 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466645 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466648 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466651 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466656 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466659 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466662 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466665 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466668 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466671 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466674 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466677 2565 flags.go:64] FLAG: --kube-reserved="" Jul 10 22:22:17.473585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466680 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466685 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466689 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466691 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466695 2565 flags.go:64] FLAG: --lock-file="" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466698 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466701 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466705 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466711 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466714 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466717 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466720 2565 flags.go:64] FLAG: --logging-format="text" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466723 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466727 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466730 2565 flags.go:64] FLAG: --manifest-url="" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466733 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466737 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466742 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466746 2565 flags.go:64] FLAG: --max-pods="110" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466749 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466753 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466756 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466759 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466763 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466766 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 22:22:17.474226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466770 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466778 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466782 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466785 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466789 2565 flags.go:64] FLAG: --pod-cidr="" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466792 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466797 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466800 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466805 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466808 2565 flags.go:64] FLAG: --port="10250" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466812 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466816 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0943b85415f161df3" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466819 2565 flags.go:64] FLAG: --qos-reserved="" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466822 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466826 2565 flags.go:64] FLAG: --register-node="true" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466829 2565 flags.go:64] FLAG: --register-with-taints="" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466833 2565 flags.go:64] FLAG: --registry-burst="10" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466836 2565 flags.go:64] FLAG: --registry-qps="5" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466839 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466842 2565 flags.go:64] FLAG: --reserved-memory="" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466846 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466849 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466853 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466856 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466859 2565 flags.go:64] FLAG: --runonce="false" Jul 10 22:22:17.474911 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466864 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466886 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466890 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466893 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466896 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466899 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466903 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466907 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466910 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466914 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466917 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466920 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466924 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466927 2565 flags.go:64] FLAG: --system-cgroups="" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466930 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466938 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466941 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466944 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466949 2565 flags.go:64] FLAG: --tls-min-version="" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466952 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466955 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466982 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466987 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466991 2565 flags.go:64] FLAG: --v="2" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.466996 2565 flags.go:64] FLAG: --version="false" Jul 10 22:22:17.475566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.467001 2565 flags.go:64] FLAG: --vmodule="" Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.467005 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.467008 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467140 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467145 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467148 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467152 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467156 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467159 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467162 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467165 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467168 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467171 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467174 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467177 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467180 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467183 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467186 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467190 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467193 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:22:17.476236 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467195 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467198 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467202 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467205 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467208 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467211 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467214 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467216 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467219 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467222 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467225 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467228 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467230 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467234 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467237 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467240 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467242 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467245 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467248 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467252 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:22:17.476758 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467255 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467258 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467261 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467264 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467266 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467269 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467272 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467275 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467278 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467281 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467284 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467286 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467289 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467292 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467296 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467298 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467301 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467304 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467307 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467309 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:22:17.477354 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467312 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467315 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467317 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467320 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467323 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467326 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467329 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467331 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467334 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467337 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467339 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467344 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467347 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467350 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467353 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467356 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467359 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467361 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467364 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467367 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:22:17.477910 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467370 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467373 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467382 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467385 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467388 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467391 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467394 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467397 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467400 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467404 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467408 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467411 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467414 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467419 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467422 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467425 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467428 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467433 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467437 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:22:17.478487 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467440 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467443 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.467446 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.467453 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.474826 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.474841 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.474857 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.474866 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474945 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474952 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474956 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474959 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474963 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474966 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474969 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:22:17.478996 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474973 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474977 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474980 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474984 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474987 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474990 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474993 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474996 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.474999 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475002 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475005 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475007 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475011 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475013 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475016 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475019 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475022 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475025 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475028 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:22:17.479391 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475031 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475034 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475038 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475041 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475044 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475047 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475050 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475055 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475060 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475065 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475069 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475072 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475075 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475078 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475081 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475084 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475087 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475090 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475093 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:22:17.479894 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475095 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475098 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475101 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475104 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475107 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475110 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475113 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475115 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475118 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475121 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475124 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475127 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475130 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475132 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475135 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475138 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475142 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475145 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475148 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475151 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:22:17.480383 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475154 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475156 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475160 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475162 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475165 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475168 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475171 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475173 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475176 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475179 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475182 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475185 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475188 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475190 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475193 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475196 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475200 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475204 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475207 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:22:17.480982 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475210 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475213 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475216 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475219 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475221 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475224 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475227 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475230 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475233 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475236 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475239 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475241 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475244 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475247 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475249 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:22:17.481477 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.475256 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475385 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475391 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475394 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475397 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475400 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475403 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475406 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475409 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475412 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475414 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475417 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475420 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475423 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475426 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475429 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475431 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475434 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475437 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:22:17.481885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475440 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475443 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475446 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475449 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475452 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475455 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475458 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475461 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475464 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475467 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475470 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475473 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475476 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475478 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475482 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475485 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475488 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475491 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475493 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:22:17.482378 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475496 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475499 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475501 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475504 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475507 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475510 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475513 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475515 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475518 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475521 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475524 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475526 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475529 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475532 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475535 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475538 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475540 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475543 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475546 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475550 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:22:17.482895 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475552 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475555 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475558 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475561 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475563 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475566 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475569 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475572 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475575 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475579 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475582 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475586 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475590 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475593 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475597 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475600 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475603 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475605 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475608 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475611 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:22:17.483412 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475614 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475618 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475625 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475629 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475633 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475636 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475639 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475642 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475645 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475648 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475651 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475654 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475657 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475660 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475663 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475666 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475669 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475672 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475675 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:22:17.483945 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475677 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:22:17.484730 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475680 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:22:17.484730 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:17.475682 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:22:17.484730 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.475693 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 22:22:17.484730 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.475806 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 22:22:17.484730 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.478125 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 22:22:17.484730 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.479571 2565 server.go:1019] "Starting client certificate rotation" Jul 10 22:22:17.484730 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.479672 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 22:22:17.484730 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.479711 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 22:22:17.484730 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.482600 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 22:22:17.487405 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.487382 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 22:22:17.498444 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.498424 2565 log.go:25] "Validated CRI v1 runtime API" Jul 10 22:22:17.500746 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.500729 2565 log.go:25] "Validated CRI v1 image API" Jul 10 22:22:17.501698 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.501679 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 22:22:17.504371 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.504350 2565 fs.go:135] Filesystem UUIDs: map[0295781e-dc2b-4092-9a9d-d7d89757d331:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dfd7a886-be72-407b-9b5e-d0fd664a0386:/dev/nvme0n1p3] Jul 10 22:22:17.504423 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.504371 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 22:22:17.510538 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.510396 2565 manager.go:216] Machine: {Timestamp:2026-07-10 22:22:17.508817009 +0000 UTC m=+1.341701386 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3134507 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2836b6b367ce9027e7fba0921f288a SystemUUID:ec2836b6-b367-ce90-27e7-fba0921f288a BootID:244a4a7c-47fa-4066-8532-48b021286991 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9e:07:f8:19:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9e:07:f8:19:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:2f:bd:08:98:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 22:22:17.510538 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.510526 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 22:22:17.510691 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.510628 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 22:22:17.512994 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.512973 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 22:22:17.513771 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.513743 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 22:22:17.513948 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.513775 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-68.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 22:22:17.515258 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.515246 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 22:22:17.515258 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.515260 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 22:22:17.515347 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.515269 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 22:22:17.516170 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.516158 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 22:22:17.517054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.517044 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 22:22:17.520784 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.520771 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 10 22:22:17.520938 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.520927 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 22:22:17.520982 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.520961 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 10 22:22:17.520982 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.520972 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 22:22:17.521039 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.520989 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 22:22:17.521039 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.520999 2565 kubelet.go:398] "Adding apiserver pod source" Jul 10 22:22:17.521039 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.521023 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 22:22:17.521559 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.521545 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 22:22:17.521631 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.521566 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 22:22:17.524072 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.524046 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 22:22:17.526149 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.526135 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 22:22:17.526211 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.526167 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 22:22:17.527384 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.527368 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 22:22:17.527443 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.527390 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 22:22:17.527443 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.527397 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 22:22:17.527443 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.527403 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 22:22:17.527443 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.527412 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 22:22:17.527443 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.527419 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 22:22:17.527443 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.527427 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 22:22:17.527443 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.527433 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 22:22:17.527443 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.527442 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 22:22:17.527443 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.527448 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 22:22:17.527705 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.527458 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 22:22:17.527705 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.527464 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 22:22:17.528627 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.528609 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 22:22:17.528627 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.528626 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 22:22:17.532484 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.532462 2565 server.go:1268] "Started kubelet" Jul 10 22:22:17.532590 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.532565 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-68.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 22:22:17.532753 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.532719 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 22:22:17.532832 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.532802 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 22:22:17.532924 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.532879 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 22:22:17.533003 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.532932 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 22:22:17.533003 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.532960 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 22:22:17.533442 ip-10-0-136-68 systemd[1]: Started Kubernetes Kubelet. Jul 10 22:22:17.534458 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.534433 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 22:22:17.537108 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.537084 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 10 22:22:17.538627 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.538608 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qggp4" Jul 10 22:22:17.539149 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.539131 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 22:22:17.540824 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.539834 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-68.ec2.internal.18c10dd5619f00f9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-68.ec2.internal,UID:ip-10-0-136-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-68.ec2.internal,},FirstTimestamp:2026-07-10 22:22:17.532432633 +0000 UTC m=+1.365317010,LastTimestamp:2026-07-10 22:22:17.532432633 +0000 UTC m=+1.365317010,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-68.ec2.internal,}" Jul 10 22:22:17.542029 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.542011 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 22:22:17.542121 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.542035 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 22:22:17.542212 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.542189 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:17.542668 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.542645 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 22:22:17.542668 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.542656 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 22:22:17.542876 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.542684 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 22:22:17.542876 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.542744 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 10 22:22:17.542876 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.542766 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 10 22:22:17.543148 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.543129 2565 factory.go:153] Registering CRI-O factory Jul 10 22:22:17.543219 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.543152 2565 factory.go:223] Registration of the crio container factory successfully Jul 10 22:22:17.543219 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.543202 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 22:22:17.543219 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.543210 2565 factory.go:55] Registering systemd factory Jul 10 22:22:17.543219 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.543218 2565 factory.go:223] Registration of the systemd container factory successfully Jul 10 22:22:17.543398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.543240 2565 factory.go:103] Registering Raw factory Jul 10 22:22:17.543398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.543254 2565 manager.go:1196] Started watching for new ooms in manager Jul 10 22:22:17.545011 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.544994 2565 manager.go:349] Starting recovery of all containers Jul 10 22:22:17.549660 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.549636 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qggp4" Jul 10 22:22:17.552119 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.552090 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 22:22:17.552275 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.552250 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 22:22:17.556830 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.556814 2565 manager.go:354] Recovery completed Jul 10 22:22:17.561842 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.561825 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:22:17.563750 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.563733 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:22:17.563812 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.563763 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:22:17.563812 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.563774 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:22:17.564392 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.564375 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 22:22:17.564392 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.564385 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:22:17.564392 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.564391 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 22:22:17.564526 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.564410 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 10 22:22:17.565155 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.565138 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:22:17.565237 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.565173 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:22:17.565237 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.565191 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:22:17.565921 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.565908 2565 policy_none.go:49] "None policy: Start" Jul 10 22:22:17.565984 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.565926 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 22:22:17.565984 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.565937 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 22:22:17.567282 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.567166 2565 policy_none.go:47] "Start" Jul 10 22:22:17.606542 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.606522 2565 manager.go:337] "Starting Device Plugin manager" Jul 10 22:22:17.606658 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.606564 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 22:22:17.606658 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.606578 2565 server.go:87] "Starting device plugin registration server" Jul 10 22:22:17.606880 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.606859 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 22:22:17.606971 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.606894 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 22:22:17.607028 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.607013 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 22:22:17.607117 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.607104 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 22:22:17.607167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.607118 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 22:22:17.607590 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.607569 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 22:22:17.607661 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.607618 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:17.683271 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.683229 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 22:22:17.684584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.684431 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 22:22:17.684584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.684468 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 22:22:17.684584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.684495 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 22:22:17.684584 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.684531 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 22:22:17.687549 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.687528 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:22:17.707041 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.707009 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:22:17.708340 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.708324 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:22:17.708437 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.708359 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:22:17.708437 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.708377 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:22:17.708437 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.708411 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.717461 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.717414 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.717461 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.717436 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-68.ec2.internal\": node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:17.732899 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.732875 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:17.785245 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.785211 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal"] Jul 10 22:22:17.785362 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.785290 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:22:17.787183 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.787165 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:22:17.787274 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.787208 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:22:17.787274 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.787223 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:22:17.788827 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.788795 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:22:17.789002 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.788987 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.789044 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.789022 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:22:17.789584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.789570 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:22:17.789671 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.789592 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:22:17.789671 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.789601 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:22:17.789671 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.789614 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:22:17.789671 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.789615 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:22:17.789671 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.789626 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:22:17.791165 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.791149 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.791216 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.791184 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:22:17.791890 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.791870 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:22:17.791987 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.791901 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:22:17.791987 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.791915 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:22:17.822941 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.822914 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-68.ec2.internal\" not found" node="ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.827900 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.827855 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-68.ec2.internal\" not found" node="ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.833379 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.833360 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:17.845187 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.845162 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec63ca07b383a4359b0dcb3157d0357a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal\" (UID: \"ec63ca07b383a4359b0dcb3157d0357a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.845271 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.845190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec63ca07b383a4359b0dcb3157d0357a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal\" (UID: \"ec63ca07b383a4359b0dcb3157d0357a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.845271 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.845210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a636bbdd5023b50004b217a7742580f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-68.ec2.internal\" (UID: \"5a636bbdd5023b50004b217a7742580f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.934392 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:17.934358 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:17.945534 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.945507 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec63ca07b383a4359b0dcb3157d0357a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal\" (UID: \"ec63ca07b383a4359b0dcb3157d0357a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.945589 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.945543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec63ca07b383a4359b0dcb3157d0357a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal\" (UID: \"ec63ca07b383a4359b0dcb3157d0357a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.945589 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.945563 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a636bbdd5023b50004b217a7742580f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-68.ec2.internal\" (UID: \"5a636bbdd5023b50004b217a7742580f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.945659 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.945621 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec63ca07b383a4359b0dcb3157d0357a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal\" (UID: \"ec63ca07b383a4359b0dcb3157d0357a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.945694 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.945680 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a636bbdd5023b50004b217a7742580f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-68.ec2.internal\" (UID: \"5a636bbdd5023b50004b217a7742580f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal" Jul 10 22:22:17.945728 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:17.945719 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec63ca07b383a4359b0dcb3157d0357a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal\" (UID: \"ec63ca07b383a4359b0dcb3157d0357a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" Jul 10 22:22:18.034670 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:18.034584 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:18.131461 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.131426 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal" Jul 10 22:22:18.131568 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.131551 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" Jul 10 22:22:18.135770 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:18.135625 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:18.236876 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:18.236836 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:18.337134 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:18.337057 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:18.432646 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.432615 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:22:18.437177 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:18.437157 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:18.480873 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.480847 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 22:22:18.481428 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.481000 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 22:22:18.481428 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.481054 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 22:22:18.538386 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:18.538355 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:18.563395 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.563364 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 22:17:17 +0000 UTC" deadline="2027-12-12 18:55:04.136575413 +0000 UTC" Jul 10 22:22:18.563395 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.563392 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12476h32m45.573186335s" Jul 10 22:22:18.639603 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:18.639522 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:18.740601 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:18.740562 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found" Jul 10 22:22:18.748657 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.748627 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:22:18.774479 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:18.774445 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a636bbdd5023b50004b217a7742580f.slice/crio-947f3726c319d4db8675b3e71130d47c71b3e8838a67e07aa76bbd567a7b8eb4 WatchSource:0}: Error finding container 947f3726c319d4db8675b3e71130d47c71b3e8838a67e07aa76bbd567a7b8eb4: Status 404 returned error can't find the container with id 947f3726c319d4db8675b3e71130d47c71b3e8838a67e07aa76bbd567a7b8eb4 Jul 10 22:22:18.775065 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:18.775038 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec63ca07b383a4359b0dcb3157d0357a.slice/crio-87032712ca5d2627723dc8801a67eee981416e8670cbb2f67ce004d6bcaaab3c WatchSource:0}: Error finding container 87032712ca5d2627723dc8801a67eee981416e8670cbb2f67ce004d6bcaaab3c: Status 404 returned error can't find the container with id 87032712ca5d2627723dc8801a67eee981416e8670cbb2f67ce004d6bcaaab3c Jul 10 22:22:18.779522 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.779505 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 22:22:18.844298 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.844260 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" Jul 10 22:22:18.856038 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.856005 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 22:22:18.857975 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.857961 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal" Jul 10 22:22:18.866189 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:18.866172 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 22:22:19.037172 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.037092 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:22:19.521695 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.521613 2565 apiserver.go:52] "Watching apiserver" Jul 10 22:22:19.530843 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.530800 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 22:22:19.531410 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.531384 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-ln9m5","openshift-multus/multus-dwzdh","openshift-multus/network-metrics-daemon-nwdcr","openshift-network-operator/iptables-alerter-pnk69","kube-system/konnectivity-agent-95zkp","kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk","openshift-cluster-node-tuning-operator/tuned-st2pb","openshift-dns/node-resolver-g96gd","openshift-network-diagnostics/network-check-target-vzlrd","openshift-ovn-kubernetes/ovnkube-node-bwjng","kube-system/global-pull-secret-syncer-b4thn","openshift-image-registry/node-ca-5zb6h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal"] Jul 10 22:22:19.533991 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.533261 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.534726 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.534646 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5zb6h" Jul 10 22:22:19.536172 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.536143 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 22:22:19.536291 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.536269 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 22:22:19.536364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.536287 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 22:22:19.536364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.536326 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rmg5q\"" Jul 10 22:22:19.536476 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.536401 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 22:22:19.536533 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.536475 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:19.536602 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:19.536567 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:19.536723 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.536627 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 22:22:19.537876 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.536865 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 22:22:19.537876 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.537211 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 22:22:19.537876 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.537256 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 22:22:19.538098 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.537908 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pnk69" Jul 10 22:22:19.539954 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.538737 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ctbdd\"" Jul 10 22:22:19.539954 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.539388 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g96gd" Jul 10 22:22:19.541091 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.541052 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.541091 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.541090 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 22:22:19.541509 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.541393 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 22:22:19.541636 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.541538 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v6gqb\"" Jul 10 22:22:19.541703 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.541655 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 22:22:19.542105 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.542023 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 22:22:19.542243 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.542225 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qwjv2\"" Jul 10 22:22:19.542442 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.542425 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 22:22:19.542661 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.542640 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.543357 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.543339 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 22:22:19.543464 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.543378 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dcr6m\"" Jul 10 22:22:19.546129 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.543626 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 22:22:19.546129 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.544204 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 22:22:19.546129 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.544270 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-95zkp" Jul 10 22:22:19.546129 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.545735 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:19.546129 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:19.545888 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:19.546533 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.546495 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 22:22:19.548424 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.546795 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-skc8r\"" Jul 10 22:22:19.548424 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.547009 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 22:22:19.548424 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.547159 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5frjd\"" Jul 10 22:22:19.548424 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.547266 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 22:22:19.548424 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.547330 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 22:22:19.548424 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.546797 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 22:22:19.548424 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.547727 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.550292 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.549329 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:19.550292 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:19.549414 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:19.550292 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.550061 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 22:22:19.550292 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.550286 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.552243 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552094 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 22:22:19.552243 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552174 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-kubernetes\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.552243 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552231 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-lib-modules\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.552467 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552257 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-var-lib-kubelet\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.552467 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552282 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7g4j\" (UniqueName: \"kubernetes.io/projected/60cf880a-7e60-46a1-90bd-3539c28c5a1e-kube-api-access-g7g4j\") pod \"iptables-alerter-pnk69\" (UID: \"60cf880a-7e60-46a1-90bd-3539c28c5a1e\") " pod="openshift-network-operator/iptables-alerter-pnk69" Jul 10 22:22:19.552467 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552304 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0fd957-b6e9-4ba3-9d5d-8ed753694c91-tmp-dir\") pod \"node-resolver-g96gd\" (UID: \"9f0fd957-b6e9-4ba3-9d5d-8ed753694c91\") " pod="openshift-dns/node-resolver-g96gd" Jul 10 22:22:19.552467 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f578f81-ae6f-44cf-8e1f-746894c94c69-tmp\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.552467 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552389 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-cnibin\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.552467 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f0fd957-b6e9-4ba3-9d5d-8ed753694c91-hosts-file\") pod \"node-resolver-g96gd\" (UID: \"9f0fd957-b6e9-4ba3-9d5d-8ed753694c91\") " pod="openshift-dns/node-resolver-g96gd" Jul 10 22:22:19.552793 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552475 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-sysctl-conf\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.552793 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-os-release\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.552793 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552536 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.552793 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552569 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzpwb\" (UniqueName: \"kubernetes.io/projected/93168b90-d2d0-4adb-8b3f-e825f87989cc-kube-api-access-hzpwb\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.552793 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552597 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47tlr\" (UniqueName: \"kubernetes.io/projected/897ef813-52ab-407d-8429-2656360c5160-kube-api-access-47tlr\") pod \"network-metrics-daemon-nwdcr\" (UID: \"897ef813-52ab-407d-8429-2656360c5160\") " pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:19.552793 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552639 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5b2\" (UniqueName: \"kubernetes.io/projected/9f0fd957-b6e9-4ba3-9d5d-8ed753694c91-kube-api-access-gg5b2\") pod \"node-resolver-g96gd\" (UID: \"9f0fd957-b6e9-4ba3-9d5d-8ed753694c91\") " pod="openshift-dns/node-resolver-g96gd" Jul 10 22:22:19.552793 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-sysconfig\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.552793 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552687 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-sysctl-d\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.552793 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-run\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.552793 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552762 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7fr5\" (UniqueName: \"kubernetes.io/projected/3f578f81-ae6f-44cf-8e1f-746894c94c69-kube-api-access-d7fr5\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552805 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552892 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee84dbb8-64c7-46d8-840a-94cc40156e29-serviceca\") pod \"node-ca-5zb6h\" (UID: \"ee84dbb8-64c7-46d8-840a-94cc40156e29\") " pod="openshift-image-registry/node-ca-5zb6h" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552928 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs\") pod \"network-metrics-daemon-nwdcr\" (UID: \"897ef813-52ab-407d-8429-2656360c5160\") " pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552954 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-modprobe-d\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.552977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-host\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553010 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnjqt\" (UniqueName: \"kubernetes.io/projected/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-kube-api-access-pnjqt\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553114 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-socket-dir\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553142 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee84dbb8-64c7-46d8-840a-94cc40156e29-host\") pod \"node-ca-5zb6h\" (UID: \"ee84dbb8-64c7-46d8-840a-94cc40156e29\") " pod="openshift-image-registry/node-ca-5zb6h" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-system-cni-dir\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553192 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553219 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553246 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60cf880a-7e60-46a1-90bd-3539c28c5a1e-host-slash\") pod \"iptables-alerter-pnk69\" (UID: \"60cf880a-7e60-46a1-90bd-3539c28c5a1e\") " pod="openshift-network-operator/iptables-alerter-pnk69" Jul 10 22:22:19.553321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553284 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-sys-fs\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.554263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/428ea365-0919-4562-bd39-f10dfdcda1d5-konnectivity-ca\") pod \"konnectivity-agent-95zkp\" (UID: \"428ea365-0919-4562-bd39-f10dfdcda1d5\") " pod="kube-system/konnectivity-agent-95zkp" Jul 10 22:22:19.554263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gg8c\" (UniqueName: \"kubernetes.io/projected/ee84dbb8-64c7-46d8-840a-94cc40156e29-kube-api-access-8gg8c\") pod \"node-ca-5zb6h\" (UID: \"ee84dbb8-64c7-46d8-840a-94cc40156e29\") " pod="openshift-image-registry/node-ca-5zb6h" Jul 10 22:22:19.554263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553434 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-systemd\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.554263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553518 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-sys\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.554263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/60cf880a-7e60-46a1-90bd-3539c28c5a1e-iptables-alerter-script\") pod \"iptables-alerter-pnk69\" (UID: \"60cf880a-7e60-46a1-90bd-3539c28c5a1e\") " pod="openshift-network-operator/iptables-alerter-pnk69" Jul 10 22:22:19.554263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553582 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-registration-dir\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.554263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553598 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jmgpx\"" Jul 10 22:22:19.554263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553635 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.554263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553665 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-tuned\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.554263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553686 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-device-dir\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.554263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553716 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/428ea365-0919-4562-bd39-f10dfdcda1d5-agent-certs\") pod \"konnectivity-agent-95zkp\" (UID: \"428ea365-0919-4562-bd39-f10dfdcda1d5\") " pod="kube-system/konnectivity-agent-95zkp" Jul 10 22:22:19.554263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553917 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 22:22:19.554263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.553409 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 22:22:19.554971 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.554331 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rck64\"" Jul 10 22:22:19.554971 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.554729 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 22:22:19.554971 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.554738 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 22:22:19.556828 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.556806 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 22:22:19.609381 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.609346 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9rpqd" Jul 10 22:22:19.617639 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.617118 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9rpqd" Jul 10 22:22:19.644356 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.644261 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-registration-dir\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654304 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-var-lib-cni-bin\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-var-lib-kubelet\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654358 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-run-multus-certs\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-node-log\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654410 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-host\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654436 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjqt\" (UniqueName: \"kubernetes.io/projected/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-kube-api-access-pnjqt\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/428ea365-0919-4562-bd39-f10dfdcda1d5-konnectivity-ca\") pod \"konnectivity-agent-95zkp\" (UID: \"428ea365-0919-4562-bd39-f10dfdcda1d5\") " pod="kube-system/konnectivity-agent-95zkp" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654487 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-multus-socket-dir-parent\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654510 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-var-lib-cni-multus\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14b9b95f-6755-4116-bf6a-317e009798e2-dbus\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654597 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-registration-dir\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654611 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee84dbb8-64c7-46d8-840a-94cc40156e29-host\") pod \"node-ca-5zb6h\" (UID: \"ee84dbb8-64c7-46d8-840a-94cc40156e29\") " pod="openshift-image-registry/node-ca-5zb6h" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-sys-fs\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-host\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654686 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-ovnkube-script-lib\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.655414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654713 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gg8c\" (UniqueName: \"kubernetes.io/projected/ee84dbb8-64c7-46d8-840a-94cc40156e29-kube-api-access-8gg8c\") pod \"node-ca-5zb6h\" (UID: \"ee84dbb8-64c7-46d8-840a-94cc40156e29\") " pod="openshift-image-registry/node-ca-5zb6h" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-cnibin\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654762 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-log-socket\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654785 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgm5r\" (UniqueName: \"kubernetes.io/projected/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-kube-api-access-wgm5r\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-tuned\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-device-dir\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654948 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-kubelet\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654974 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-kubernetes\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.654997 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-hostroot\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655061 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-etc-kubernetes\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655109 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-etc-openvswitch\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655140 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-cnibin\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655227 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-system-cni-dir\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-sysctl-conf\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655297 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-multus-cni-dir\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655322 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-os-release\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655416 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-run-k8s-cni-cncf-io\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.656364 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-cni-bin\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655474 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47tlr\" (UniqueName: \"kubernetes.io/projected/897ef813-52ab-407d-8429-2656360c5160-kube-api-access-47tlr\") pod \"network-metrics-daemon-nwdcr\" (UID: \"897ef813-52ab-407d-8429-2656360c5160\") " pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655501 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-sysconfig\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-run\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655613 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-cni-netd\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655642 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee84dbb8-64c7-46d8-840a-94cc40156e29-serviceca\") pod \"node-ca-5zb6h\" (UID: \"ee84dbb8-64c7-46d8-840a-94cc40156e29\") " pod="openshift-image-registry/node-ca-5zb6h" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs\") pod \"network-metrics-daemon-nwdcr\" (UID: \"897ef813-52ab-407d-8429-2656360c5160\") " pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655742 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-modprobe-d\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655771 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-socket-dir\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-run-ovn\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655922 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-system-cni-dir\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.655993 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656023 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.657319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656052 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60cf880a-7e60-46a1-90bd-3539c28c5a1e-host-slash\") pod \"iptables-alerter-pnk69\" (UID: \"60cf880a-7e60-46a1-90bd-3539c28c5a1e\") " pod="openshift-network-operator/iptables-alerter-pnk69" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656081 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-run-netns\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656109 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-var-lib-openvswitch\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656359 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-env-overrides\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656393 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-systemd\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656421 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-sys\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656451 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/60cf880a-7e60-46a1-90bd-3539c28c5a1e-iptables-alerter-script\") pod \"iptables-alerter-pnk69\" (UID: \"60cf880a-7e60-46a1-90bd-3539c28c5a1e\") " pod="openshift-network-operator/iptables-alerter-pnk69" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a79673b-9490-4771-a6ff-5dd650bf6c28-multus-daemon-config\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656517 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-slash\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656546 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/428ea365-0919-4562-bd39-f10dfdcda1d5-agent-certs\") pod \"konnectivity-agent-95zkp\" (UID: \"428ea365-0919-4562-bd39-f10dfdcda1d5\") " pod="kube-system/konnectivity-agent-95zkp" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvwsn\" (UniqueName: \"kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn\") pod \"network-check-target-vzlrd\" (UID: \"dc25ad9c-ea7d-40af-85e5-54b27aef019b\") " pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656646 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-lib-modules\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-var-lib-kubelet\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7g4j\" (UniqueName: \"kubernetes.io/projected/60cf880a-7e60-46a1-90bd-3539c28c5a1e-kube-api-access-g7g4j\") pod \"iptables-alerter-pnk69\" (UID: \"60cf880a-7e60-46a1-90bd-3539c28c5a1e\") " pod="openshift-network-operator/iptables-alerter-pnk69" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656734 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-multus-conf-dir\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656766 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4vkn\" (UniqueName: \"kubernetes.io/projected/7a79673b-9490-4771-a6ff-5dd650bf6c28-kube-api-access-p4vkn\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.658167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-run-netns\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-run-openvswitch\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0fd957-b6e9-4ba3-9d5d-8ed753694c91-tmp-dir\") pod \"node-resolver-g96gd\" (UID: \"9f0fd957-b6e9-4ba3-9d5d-8ed753694c91\") " pod="openshift-dns/node-resolver-g96gd" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656900 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f578f81-ae6f-44cf-8e1f-746894c94c69-tmp\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a79673b-9490-4771-a6ff-5dd650bf6c28-cni-binary-copy\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656960 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-systemd-units\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.656987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-run-systemd\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.657017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.657048 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-ovnkube-config\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.657076 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-ovn-node-metrics-cert\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.657106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f0fd957-b6e9-4ba3-9d5d-8ed753694c91-hosts-file\") pod \"node-resolver-g96gd\" (UID: \"9f0fd957-b6e9-4ba3-9d5d-8ed753694c91\") " pod="openshift-dns/node-resolver-g96gd" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.657133 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-os-release\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.657162 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.657191 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpwb\" (UniqueName: \"kubernetes.io/projected/93168b90-d2d0-4adb-8b3f-e825f87989cc-kube-api-access-hzpwb\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.657226 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14b9b95f-6755-4116-bf6a-317e009798e2-kubelet-config\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.657256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5b2\" (UniqueName: \"kubernetes.io/projected/9f0fd957-b6e9-4ba3-9d5d-8ed753694c91-kube-api-access-gg5b2\") pod \"node-resolver-g96gd\" (UID: \"9f0fd957-b6e9-4ba3-9d5d-8ed753694c91\") " pod="openshift-dns/node-resolver-g96gd" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.657288 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-sysctl-d\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.659054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.657328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7fr5\" (UniqueName: \"kubernetes.io/projected/3f578f81-ae6f-44cf-8e1f-746894c94c69-kube-api-access-d7fr5\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.659884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.657860 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 22:22:19.659884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.659204 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.659884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.659300 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-sys-fs\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.659884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.659346 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee84dbb8-64c7-46d8-840a-94cc40156e29-host\") pod \"node-ca-5zb6h\" (UID: \"ee84dbb8-64c7-46d8-840a-94cc40156e29\") " pod="openshift-image-registry/node-ca-5zb6h" Jul 10 22:22:19.659884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.659616 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-device-dir\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.659884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.659669 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-kubernetes\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.659884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.659713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-cnibin\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.659884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.659838 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-sysctl-conf\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.660288 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.660141 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-sysconfig\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.660288 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.660191 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-run\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.660447 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.660332 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.660447 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.660441 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.660849 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee84dbb8-64c7-46d8-840a-94cc40156e29-serviceca\") pod \"node-ca-5zb6h\" (UID: \"ee84dbb8-64c7-46d8-840a-94cc40156e29\") " pod="openshift-image-registry/node-ca-5zb6h" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.660969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/428ea365-0919-4562-bd39-f10dfdcda1d5-konnectivity-ca\") pod \"konnectivity-agent-95zkp\" (UID: \"428ea365-0919-4562-bd39-f10dfdcda1d5\") " pod="kube-system/konnectivity-agent-95zkp" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:19.661003 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:19.661071 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs podName:897ef813-52ab-407d-8429-2656360c5160 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:20.16104993 +0000 UTC m=+3.993934291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs") pod "network-metrics-daemon-nwdcr" (UID: "897ef813-52ab-407d-8429-2656360c5160") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.661186 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-modprobe-d\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.661236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.661294 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60cf880a-7e60-46a1-90bd-3539c28c5a1e-host-slash\") pod \"iptables-alerter-pnk69\" (UID: \"60cf880a-7e60-46a1-90bd-3539c28c5a1e\") " pod="openshift-network-operator/iptables-alerter-pnk69" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.661347 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-systemd\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.661386 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-sys\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.661643 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.661746 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-socket-dir\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.661802 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-system-cni-dir\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.662038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93168b90-d2d0-4adb-8b3f-e825f87989cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.662115 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f0fd957-b6e9-4ba3-9d5d-8ed753694c91-hosts-file\") pod \"node-resolver-g96gd\" (UID: \"9f0fd957-b6e9-4ba3-9d5d-8ed753694c91\") " pod="openshift-dns/node-resolver-g96gd" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.662135 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0fd957-b6e9-4ba3-9d5d-8ed753694c91-tmp-dir\") pod \"node-resolver-g96gd\" (UID: \"9f0fd957-b6e9-4ba3-9d5d-8ed753694c91\") " pod="openshift-dns/node-resolver-g96gd" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.662183 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-os-release\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.662716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.662396 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-lib-modules\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.665313 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.662495 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-sysctl-d\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.665313 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.662496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/60cf880a-7e60-46a1-90bd-3539c28c5a1e-iptables-alerter-script\") pod \"iptables-alerter-pnk69\" (UID: \"60cf880a-7e60-46a1-90bd-3539c28c5a1e\") " pod="openshift-network-operator/iptables-alerter-pnk69" Jul 10 22:22:19.665313 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.662653 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f578f81-ae6f-44cf-8e1f-746894c94c69-var-lib-kubelet\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.668625 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.668580 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f578f81-ae6f-44cf-8e1f-746894c94c69-tmp\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.669691 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.669637 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/428ea365-0919-4562-bd39-f10dfdcda1d5-agent-certs\") pod \"konnectivity-agent-95zkp\" (UID: \"428ea365-0919-4562-bd39-f10dfdcda1d5\") " pod="kube-system/konnectivity-agent-95zkp" Jul 10 22:22:19.669691 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.669675 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7fr5\" (UniqueName: \"kubernetes.io/projected/3f578f81-ae6f-44cf-8e1f-746894c94c69-kube-api-access-d7fr5\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.671275 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.671192 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjqt\" (UniqueName: \"kubernetes.io/projected/1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc-kube-api-access-pnjqt\") pod \"multus-additional-cni-plugins-ln9m5\" (UID: \"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc\") " pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.672045 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.671867 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47tlr\" (UniqueName: \"kubernetes.io/projected/897ef813-52ab-407d-8429-2656360c5160-kube-api-access-47tlr\") pod \"network-metrics-daemon-nwdcr\" (UID: \"897ef813-52ab-407d-8429-2656360c5160\") " pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:19.673341 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.673298 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gg8c\" (UniqueName: \"kubernetes.io/projected/ee84dbb8-64c7-46d8-840a-94cc40156e29-kube-api-access-8gg8c\") pod \"node-ca-5zb6h\" (UID: \"ee84dbb8-64c7-46d8-840a-94cc40156e29\") " pod="openshift-image-registry/node-ca-5zb6h" Jul 10 22:22:19.673493 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.673470 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f578f81-ae6f-44cf-8e1f-746894c94c69-etc-tuned\") pod \"tuned-st2pb\" (UID: \"3f578f81-ae6f-44cf-8e1f-746894c94c69\") " pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.676243 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.676219 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7g4j\" (UniqueName: \"kubernetes.io/projected/60cf880a-7e60-46a1-90bd-3539c28c5a1e-kube-api-access-g7g4j\") pod \"iptables-alerter-pnk69\" (UID: \"60cf880a-7e60-46a1-90bd-3539c28c5a1e\") " pod="openshift-network-operator/iptables-alerter-pnk69" Jul 10 22:22:19.676345 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.676255 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5b2\" (UniqueName: \"kubernetes.io/projected/9f0fd957-b6e9-4ba3-9d5d-8ed753694c91-kube-api-access-gg5b2\") pod \"node-resolver-g96gd\" (UID: \"9f0fd957-b6e9-4ba3-9d5d-8ed753694c91\") " pod="openshift-dns/node-resolver-g96gd" Jul 10 22:22:19.678678 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.678642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzpwb\" (UniqueName: \"kubernetes.io/projected/93168b90-d2d0-4adb-8b3f-e825f87989cc-kube-api-access-hzpwb\") pod \"aws-ebs-csi-driver-node-7r2jk\" (UID: \"93168b90-d2d0-4adb-8b3f-e825f87989cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.691640 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.691532 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal" event={"ID":"5a636bbdd5023b50004b217a7742580f","Type":"ContainerStarted","Data":"947f3726c319d4db8675b3e71130d47c71b3e8838a67e07aa76bbd567a7b8eb4"} Jul 10 22:22:19.694100 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.694072 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" event={"ID":"ec63ca07b383a4359b0dcb3157d0357a","Type":"ContainerStarted","Data":"87032712ca5d2627723dc8801a67eee981416e8670cbb2f67ce004d6bcaaab3c"} Jul 10 22:22:19.758134 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-system-cni-dir\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.758134 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758143 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-multus-cni-dir\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.758398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-os-release\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.758398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-run-k8s-cni-cncf-io\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.758398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758227 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-cni-bin\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-cni-netd\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758296 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-run-ovn\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758321 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:19.758398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758348 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-run-netns\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.758398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-var-lib-openvswitch\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758411 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-env-overrides\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758437 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a79673b-9490-4771-a6ff-5dd650bf6c28-multus-daemon-config\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-slash\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758474 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758491 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwsn\" (UniqueName: \"kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn\") pod \"network-check-target-vzlrd\" (UID: \"dc25ad9c-ea7d-40af-85e5-54b27aef019b\") " pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-multus-conf-dir\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758529 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4vkn\" (UniqueName: \"kubernetes.io/projected/7a79673b-9490-4771-a6ff-5dd650bf6c28-kube-api-access-p4vkn\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758554 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-run-netns\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758581 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-run-openvswitch\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a79673b-9490-4771-a6ff-5dd650bf6c28-cni-binary-copy\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-systemd-units\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758656 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-run-systemd\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758710 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-ovnkube-config\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758737 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-ovn-node-metrics-cert\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14b9b95f-6755-4116-bf6a-317e009798e2-kubelet-config\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:19.758814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758798 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-var-lib-cni-bin\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758824 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-var-lib-kubelet\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-run-multus-certs\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758894 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-node-log\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-multus-socket-dir-parent\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758951 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-var-lib-cni-multus\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.758980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14b9b95f-6755-4116-bf6a-317e009798e2-dbus\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759009 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-ovnkube-script-lib\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759037 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-cnibin\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759061 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-log-socket\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759087 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgm5r\" (UniqueName: \"kubernetes.io/projected/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-kube-api-access-wgm5r\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-kubelet\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759159 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-hostroot\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759186 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-etc-kubernetes\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759218 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-etc-openvswitch\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759315 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-etc-openvswitch\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759339 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-env-overrides\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759379 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-systemd-units\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.761981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759425 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-system-cni-dir\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759432 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-run-systemd\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759481 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a79673b-9490-4771-a6ff-5dd650bf6c28-cni-binary-copy\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759486 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759572 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-multus-cni-dir\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759646 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-os-release\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759691 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-run-k8s-cni-cncf-io\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759741 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-cni-bin\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759785 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-cni-netd\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.759827 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-run-ovn\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:19.759936 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:19.760005 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret podName:14b9b95f-6755-4116-bf6a-317e009798e2 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:20.259984554 +0000 UTC m=+4.092868938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret") pod "global-pull-secret-syncer-b4thn" (UID: "14b9b95f-6755-4116-bf6a-317e009798e2") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760010 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a79673b-9490-4771-a6ff-5dd650bf6c28-multus-daemon-config\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760032 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-ovnkube-config\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760091 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760097 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-slash\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760154 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-multus-conf-dir\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760248 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-run-netns\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.762902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760309 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-var-lib-openvswitch\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760359 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-run-netns\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760411 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-run-openvswitch\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760455 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-var-lib-cni-bin\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760498 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-var-lib-kubelet\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760513 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14b9b95f-6755-4116-bf6a-317e009798e2-dbus\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760541 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-run-multus-certs\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760570 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14b9b95f-6755-4116-bf6a-317e009798e2-kubelet-config\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-node-log\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760621 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-cnibin\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760663 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-multus-socket-dir-parent\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-hostroot\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760863 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-host-kubelet\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760925 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-etc-kubernetes\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760937 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a79673b-9490-4771-a6ff-5dd650bf6c28-host-var-lib-cni-multus\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.760970 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-log-socket\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.763844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.761154 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-ovnkube-script-lib\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.764665 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.764092 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-ovn-node-metrics-cert\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.769373 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:19.769318 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:22:19.769373 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:19.769338 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:22:19.769373 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:19.769352 2565 projected.go:196] Error preparing data for projected volume kube-api-access-fvwsn for pod openshift-network-diagnostics/network-check-target-vzlrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:19.769611 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:19.769429 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn podName:dc25ad9c-ea7d-40af-85e5-54b27aef019b nodeName:}" failed. No retries permitted until 2026-07-10 22:22:20.269410101 +0000 UTC m=+4.102294475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fvwsn" (UniqueName: "kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn") pod "network-check-target-vzlrd" (UID: "dc25ad9c-ea7d-40af-85e5-54b27aef019b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:19.771041 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.770992 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4vkn\" (UniqueName: \"kubernetes.io/projected/7a79673b-9490-4771-a6ff-5dd650bf6c28-kube-api-access-p4vkn\") pod \"multus-dwzdh\" (UID: \"7a79673b-9490-4771-a6ff-5dd650bf6c28\") " pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.771736 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.771681 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgm5r\" (UniqueName: \"kubernetes.io/projected/4c95ccb2-4649-40f6-ae4b-9cdd312797ec-kube-api-access-wgm5r\") pod \"ovnkube-node-bwjng\" (UID: \"4c95ccb2-4649-40f6-ae4b-9cdd312797ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.852738 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.852447 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ln9m5" Jul 10 22:22:19.865074 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.865044 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5zb6h" Jul 10 22:22:19.876207 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.876179 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pnk69" Jul 10 22:22:19.879851 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:19.879802 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee84dbb8_64c7_46d8_840a_94cc40156e29.slice/crio-df1cf80484da55a6d411d3e0443ed6dd5e138045806471664d17abd6824536ce WatchSource:0}: Error finding container df1cf80484da55a6d411d3e0443ed6dd5e138045806471664d17abd6824536ce: Status 404 returned error can't find the container with id df1cf80484da55a6d411d3e0443ed6dd5e138045806471664d17abd6824536ce Jul 10 22:22:19.885979 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.885724 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g96gd" Jul 10 22:22:19.887206 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:19.887171 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60cf880a_7e60_46a1_90bd_3539c28c5a1e.slice/crio-fb85b732561fa4e789d4fb41de16573f3467b3ef7b13faa2f7725b9de4f88ef3 WatchSource:0}: Error finding container fb85b732561fa4e789d4fb41de16573f3467b3ef7b13faa2f7725b9de4f88ef3: Status 404 returned error can't find the container with id fb85b732561fa4e789d4fb41de16573f3467b3ef7b13faa2f7725b9de4f88ef3 Jul 10 22:22:19.894926 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.894905 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-st2pb" Jul 10 22:22:19.898389 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:19.898337 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0fd957_b6e9_4ba3_9d5d_8ed753694c91.slice/crio-f8b86db5c608ec3ebbf110e1f6c24b213055a186157f66fd033489250087c2a0 WatchSource:0}: Error finding container f8b86db5c608ec3ebbf110e1f6c24b213055a186157f66fd033489250087c2a0: Status 404 returned error can't find the container with id f8b86db5c608ec3ebbf110e1f6c24b213055a186157f66fd033489250087c2a0 Jul 10 22:22:19.904500 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:19.904465 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f578f81_ae6f_44cf_8e1f_746894c94c69.slice/crio-00789ce7ab5a6411f3f50bd88f00dccdce3b301c5992035ba2e7e8954c9a32c5 WatchSource:0}: Error finding container 00789ce7ab5a6411f3f50bd88f00dccdce3b301c5992035ba2e7e8954c9a32c5: Status 404 returned error can't find the container with id 00789ce7ab5a6411f3f50bd88f00dccdce3b301c5992035ba2e7e8954c9a32c5 Jul 10 22:22:19.905299 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.905272 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" Jul 10 22:22:19.914010 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.913986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-95zkp" Jul 10 22:22:19.914813 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:19.914786 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93168b90_d2d0_4adb_8b3f_e825f87989cc.slice/crio-9a0509118e6ef58af7b06f4686d8b82a14665b53f1dcee65f6a1e0cc2d1735bd WatchSource:0}: Error finding container 9a0509118e6ef58af7b06f4686d8b82a14665b53f1dcee65f6a1e0cc2d1735bd: Status 404 returned error can't find the container with id 9a0509118e6ef58af7b06f4686d8b82a14665b53f1dcee65f6a1e0cc2d1735bd Jul 10 22:22:19.923268 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.923101 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:19.923453 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:19.923433 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428ea365_0919_4562_bd39_f10dfdcda1d5.slice/crio-8dad66d1633ccf5b9438087be3dd46b13ddbc691a0ca00c01221035af6179123 WatchSource:0}: Error finding container 8dad66d1633ccf5b9438087be3dd46b13ddbc691a0ca00c01221035af6179123: Status 404 returned error can't find the container with id 8dad66d1633ccf5b9438087be3dd46b13ddbc691a0ca00c01221035af6179123 Jul 10 22:22:19.928989 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:19.928967 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dwzdh" Jul 10 22:22:19.934347 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:19.934322 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c95ccb2_4649_40f6_ae4b_9cdd312797ec.slice/crio-3358b9728d321965c0fe0c53d13dcc346c043ccafb9a1a5fd27895c5baf01f7f WatchSource:0}: Error finding container 3358b9728d321965c0fe0c53d13dcc346c043ccafb9a1a5fd27895c5baf01f7f: Status 404 returned error can't find the container with id 3358b9728d321965c0fe0c53d13dcc346c043ccafb9a1a5fd27895c5baf01f7f Jul 10 22:22:19.940885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:19.940859 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a79673b_9490_4771_a6ff_5dd650bf6c28.slice/crio-308b6550249e09f5caf4cb850b844cfb75eeb1d7115db4382a3ae7d45818a911 WatchSource:0}: Error finding container 308b6550249e09f5caf4cb850b844cfb75eeb1d7115db4382a3ae7d45818a911: Status 404 returned error can't find the container with id 308b6550249e09f5caf4cb850b844cfb75eeb1d7115db4382a3ae7d45818a911 Jul 10 22:22:20.064905 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.064821 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:22:20.162186 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.162154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs\") pod \"network-metrics-daemon-nwdcr\" (UID: \"897ef813-52ab-407d-8429-2656360c5160\") " pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:20.162375 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:20.162307 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:20.162445 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:20.162386 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs podName:897ef813-52ab-407d-8429-2656360c5160 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:21.162363215 +0000 UTC m=+4.995247589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs") pod "network-metrics-daemon-nwdcr" (UID: "897ef813-52ab-407d-8429-2656360c5160") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:20.263162 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.263127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:20.263354 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:20.263327 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:20.263429 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:20.263416 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret podName:14b9b95f-6755-4116-bf6a-317e009798e2 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:21.263395979 +0000 UTC m=+5.096280346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret") pod "global-pull-secret-syncer-b4thn" (UID: "14b9b95f-6755-4116-bf6a-317e009798e2") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:20.363635 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.363541 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwsn\" (UniqueName: \"kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn\") pod \"network-check-target-vzlrd\" (UID: \"dc25ad9c-ea7d-40af-85e5-54b27aef019b\") " pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:20.363797 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:20.363772 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:22:20.363870 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:20.363825 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:22:20.363870 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:20.363841 2565 projected.go:196] Error preparing data for projected volume kube-api-access-fvwsn for pod openshift-network-diagnostics/network-check-target-vzlrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:20.363948 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:20.363913 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn podName:dc25ad9c-ea7d-40af-85e5-54b27aef019b nodeName:}" failed. No retries permitted until 2026-07-10 22:22:21.363891189 +0000 UTC m=+5.196775567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvwsn" (UniqueName: "kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn") pod "network-check-target-vzlrd" (UID: "dc25ad9c-ea7d-40af-85e5-54b27aef019b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:20.619758 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.619601 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 22:17:19 +0000 UTC" deadline="2028-02-07 22:42:01.870212891 +0000 UTC" Jul 10 22:22:20.619758 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.619641 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13848h19m41.250577147s" Jul 10 22:22:20.682991 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.682755 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:22:20.684908 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.684880 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:20.685057 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:20.685009 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:20.698299 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.698263 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dwzdh" event={"ID":"7a79673b-9490-4771-a6ff-5dd650bf6c28","Type":"ContainerStarted","Data":"308b6550249e09f5caf4cb850b844cfb75eeb1d7115db4382a3ae7d45818a911"} Jul 10 22:22:20.700252 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.700152 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" event={"ID":"4c95ccb2-4649-40f6-ae4b-9cdd312797ec","Type":"ContainerStarted","Data":"3358b9728d321965c0fe0c53d13dcc346c043ccafb9a1a5fd27895c5baf01f7f"} Jul 10 22:22:20.701767 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.701742 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" event={"ID":"93168b90-d2d0-4adb-8b3f-e825f87989cc","Type":"ContainerStarted","Data":"9a0509118e6ef58af7b06f4686d8b82a14665b53f1dcee65f6a1e0cc2d1735bd"} Jul 10 22:22:20.703230 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.703207 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pnk69" event={"ID":"60cf880a-7e60-46a1-90bd-3539c28c5a1e","Type":"ContainerStarted","Data":"fb85b732561fa4e789d4fb41de16573f3467b3ef7b13faa2f7725b9de4f88ef3"} Jul 10 22:22:20.705571 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.705541 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5zb6h" event={"ID":"ee84dbb8-64c7-46d8-840a-94cc40156e29","Type":"ContainerStarted","Data":"df1cf80484da55a6d411d3e0443ed6dd5e138045806471664d17abd6824536ce"} Jul 10 22:22:20.707112 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.707082 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-95zkp" event={"ID":"428ea365-0919-4562-bd39-f10dfdcda1d5","Type":"ContainerStarted","Data":"8dad66d1633ccf5b9438087be3dd46b13ddbc691a0ca00c01221035af6179123"} Jul 10 22:22:20.708747 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.708648 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-st2pb" event={"ID":"3f578f81-ae6f-44cf-8e1f-746894c94c69","Type":"ContainerStarted","Data":"00789ce7ab5a6411f3f50bd88f00dccdce3b301c5992035ba2e7e8954c9a32c5"} Jul 10 22:22:20.711214 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.711184 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g96gd" event={"ID":"9f0fd957-b6e9-4ba3-9d5d-8ed753694c91","Type":"ContainerStarted","Data":"f8b86db5c608ec3ebbf110e1f6c24b213055a186157f66fd033489250087c2a0"} Jul 10 22:22:20.713018 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:20.712961 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ln9m5" event={"ID":"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc","Type":"ContainerStarted","Data":"dfc7fb9c35fe1c782b89c32bcfbce2b234f7695066146553db6a709243549612"} Jul 10 22:22:21.171105 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:21.171067 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs\") pod \"network-metrics-daemon-nwdcr\" (UID: \"897ef813-52ab-407d-8429-2656360c5160\") " pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:21.171291 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:21.171261 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:21.171355 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:21.171334 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs podName:897ef813-52ab-407d-8429-2656360c5160 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:23.171313772 +0000 UTC m=+7.004198156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs") pod "network-metrics-daemon-nwdcr" (UID: "897ef813-52ab-407d-8429-2656360c5160") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:21.271977 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:21.271882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:21.272141 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:21.272041 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:21.272141 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:21.272110 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret podName:14b9b95f-6755-4116-bf6a-317e009798e2 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:23.272089051 +0000 UTC m=+7.104973432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret") pod "global-pull-secret-syncer-b4thn" (UID: "14b9b95f-6755-4116-bf6a-317e009798e2") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:21.372783 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:21.372740 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwsn\" (UniqueName: \"kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn\") pod \"network-check-target-vzlrd\" (UID: \"dc25ad9c-ea7d-40af-85e5-54b27aef019b\") " pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:21.372986 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:21.372968 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:22:21.373055 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:21.372989 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:22:21.373055 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:21.373002 2565 projected.go:196] Error preparing data for projected volume kube-api-access-fvwsn for pod openshift-network-diagnostics/network-check-target-vzlrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:21.373157 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:21.373064 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn podName:dc25ad9c-ea7d-40af-85e5-54b27aef019b nodeName:}" failed. No retries permitted until 2026-07-10 22:22:23.3730448 +0000 UTC m=+7.205929173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvwsn" (UniqueName: "kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn") pod "network-check-target-vzlrd" (UID: "dc25ad9c-ea7d-40af-85e5-54b27aef019b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:21.620139 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:21.620023 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 22:17:19 +0000 UTC" deadline="2028-02-02 17:41:52.655129221 +0000 UTC" Jul 10 22:22:21.620139 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:21.620086 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13723h19m31.035046998s" Jul 10 22:22:21.689100 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:21.688395 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:21.689100 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:21.688525 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:21.689100 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:21.688936 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:21.689100 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:21.689042 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:22.685177 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:22.684853 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:22.685667 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:22.685240 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:22.755720 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:22.755677 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" event={"ID":"ec63ca07b383a4359b0dcb3157d0357a","Type":"ContainerStarted","Data":"76c6532094307499943415afd52550e0c5ae0354429977bb14e38f1667e15694"} Jul 10 22:22:22.761444 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:22.760842 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal" event={"ID":"5a636bbdd5023b50004b217a7742580f","Type":"ContainerStarted","Data":"8e26e5357d992b842ff749d6c6ca70a598090234b501866cc9fd2bf8ce4bffed"} Jul 10 22:22:23.190023 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:23.189995 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs\") pod \"network-metrics-daemon-nwdcr\" (UID: \"897ef813-52ab-407d-8429-2656360c5160\") " pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:23.190211 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:23.190168 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:23.190268 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:23.190226 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs podName:897ef813-52ab-407d-8429-2656360c5160 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:27.190207415 +0000 UTC m=+11.023091790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs") pod "network-metrics-daemon-nwdcr" (UID: "897ef813-52ab-407d-8429-2656360c5160") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:23.291493 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:23.291453 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:23.291680 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:23.291645 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:23.291744 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:23.291714 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret podName:14b9b95f-6755-4116-bf6a-317e009798e2 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:27.291693685 +0000 UTC m=+11.124578060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret") pod "global-pull-secret-syncer-b4thn" (UID: "14b9b95f-6755-4116-bf6a-317e009798e2") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:23.393008 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:23.392930 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwsn\" (UniqueName: \"kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn\") pod \"network-check-target-vzlrd\" (UID: \"dc25ad9c-ea7d-40af-85e5-54b27aef019b\") " pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:23.393191 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:23.393092 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:22:23.393191 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:23.393114 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:22:23.393191 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:23.393126 2565 projected.go:196] Error preparing data for projected volume kube-api-access-fvwsn for pod openshift-network-diagnostics/network-check-target-vzlrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:23.393191 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:23.393187 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn podName:dc25ad9c-ea7d-40af-85e5-54b27aef019b nodeName:}" failed. No retries permitted until 2026-07-10 22:22:27.393169864 +0000 UTC m=+11.226054230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvwsn" (UniqueName: "kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn") pod "network-check-target-vzlrd" (UID: "dc25ad9c-ea7d-40af-85e5-54b27aef019b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:23.688070 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:23.687986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:23.688518 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:23.688134 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:23.688589 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:23.688516 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:23.688650 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:23.688634 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:24.686175 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:24.685481 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:24.686175 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:24.685626 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:25.685761 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:25.685726 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:25.686267 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:25.685725 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:25.686267 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:25.685883 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:25.686267 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:25.686008 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:26.685555 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:26.685514 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:26.685764 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:26.685656 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:26.777536 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:26.777499 2565 generic.go:356] "Generic (PLEG): container finished" podID="ec63ca07b383a4359b0dcb3157d0357a" containerID="76c6532094307499943415afd52550e0c5ae0354429977bb14e38f1667e15694" exitCode=0 Jul 10 22:22:26.778006 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:26.777558 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" event={"ID":"ec63ca07b383a4359b0dcb3157d0357a","Type":"ContainerDied","Data":"76c6532094307499943415afd52550e0c5ae0354429977bb14e38f1667e15694"} Jul 10 22:22:26.792443 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:26.791942 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal" podStartSLOduration=7.791921926 podStartE2EDuration="7.791921926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:22.788904448 +0000 UTC m=+6.621788832" watchObservedRunningTime="2026-07-10 22:22:26.791921926 +0000 UTC m=+10.624806310" Jul 10 22:22:27.230890 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:27.230853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs\") pod \"network-metrics-daemon-nwdcr\" (UID: \"897ef813-52ab-407d-8429-2656360c5160\") " pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:27.231087 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:27.231069 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:27.231153 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:27.231140 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs podName:897ef813-52ab-407d-8429-2656360c5160 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:35.231119237 +0000 UTC m=+19.064003615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs") pod "network-metrics-daemon-nwdcr" (UID: "897ef813-52ab-407d-8429-2656360c5160") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:27.332149 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:27.332111 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:27.332336 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:27.332287 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:27.332408 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:27.332373 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret podName:14b9b95f-6755-4116-bf6a-317e009798e2 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:35.332351182 +0000 UTC m=+19.165235767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret") pod "global-pull-secret-syncer-b4thn" (UID: "14b9b95f-6755-4116-bf6a-317e009798e2") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:27.432926 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:27.432881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwsn\" (UniqueName: \"kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn\") pod \"network-check-target-vzlrd\" (UID: \"dc25ad9c-ea7d-40af-85e5-54b27aef019b\") " pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:27.433100 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:27.433041 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:22:27.433100 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:27.433070 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:22:27.433100 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:27.433084 2565 projected.go:196] Error preparing data for projected volume kube-api-access-fvwsn for pod openshift-network-diagnostics/network-check-target-vzlrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:27.433253 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:27.433148 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn podName:dc25ad9c-ea7d-40af-85e5-54b27aef019b nodeName:}" failed. No retries permitted until 2026-07-10 22:22:35.43313125 +0000 UTC m=+19.266015610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvwsn" (UniqueName: "kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn") pod "network-check-target-vzlrd" (UID: "dc25ad9c-ea7d-40af-85e5-54b27aef019b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:27.685940 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:27.685809 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:27.686119 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:27.685932 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:27.686119 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:27.685988 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:27.686229 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:27.686119 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:28.685411 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:28.685320 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:28.685890 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:28.685458 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:29.685219 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:29.685176 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:29.685219 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:29.685214 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:29.685466 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:29.685332 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:29.685466 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:29.685439 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:30.684946 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:30.684913 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:30.685109 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:30.685034 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:31.685719 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:31.685684 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:31.685719 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:31.685730 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:31.686241 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:31.685875 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:31.686241 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:31.685969 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:32.685161 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:32.685128 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:32.685366 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:32.685250 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:33.684976 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:33.684938 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:33.685414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:33.684986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:33.685414 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:33.685096 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:33.685414 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:33.685227 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:34.685196 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:34.685157 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:34.685665 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:34.685289 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:35.291744 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:35.291697 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs\") pod \"network-metrics-daemon-nwdcr\" (UID: \"897ef813-52ab-407d-8429-2656360c5160\") " pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:35.291945 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:35.291897 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:35.292001 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:35.291985 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs podName:897ef813-52ab-407d-8429-2656360c5160 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:51.291962307 +0000 UTC m=+35.124846668 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs") pod "network-metrics-daemon-nwdcr" (UID: "897ef813-52ab-407d-8429-2656360c5160") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:35.393016 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:35.392973 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:35.393182 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:35.393143 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:35.393245 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:35.393211 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret podName:14b9b95f-6755-4116-bf6a-317e009798e2 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:51.393196553 +0000 UTC m=+35.226080914 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret") pod "global-pull-secret-syncer-b4thn" (UID: "14b9b95f-6755-4116-bf6a-317e009798e2") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:35.493487 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:35.493447 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwsn\" (UniqueName: \"kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn\") pod \"network-check-target-vzlrd\" (UID: \"dc25ad9c-ea7d-40af-85e5-54b27aef019b\") " pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:35.493681 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:35.493657 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:22:35.493749 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:35.493692 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:22:35.493749 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:35.493708 2565 projected.go:196] Error preparing data for projected volume kube-api-access-fvwsn for pod openshift-network-diagnostics/network-check-target-vzlrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:35.493871 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:35.493777 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn podName:dc25ad9c-ea7d-40af-85e5-54b27aef019b nodeName:}" failed. No retries permitted until 2026-07-10 22:22:51.49375685 +0000 UTC m=+35.326641214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvwsn" (UniqueName: "kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn") pod "network-check-target-vzlrd" (UID: "dc25ad9c-ea7d-40af-85e5-54b27aef019b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:35.685227 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:35.685139 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:35.685660 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:35.685273 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:35.685660 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:35.685328 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:35.685660 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:35.685468 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:36.685767 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:36.685732 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:36.686196 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:36.685871 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:37.685988 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:37.685953 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:37.686414 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:37.686060 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:37.686414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:37.686173 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:37.686414 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:37.686310 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:38.684755 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:38.684717 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:38.684939 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:38.684854 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:39.685099 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.685066 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:39.685719 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:39.685211 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:39.685719 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.685255 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:39.685719 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:39.685380 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:39.804047 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.804013 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" event={"ID":"ec63ca07b383a4359b0dcb3157d0357a","Type":"ContainerStarted","Data":"cce087f99d3e0a98bdce6657da19f2539f745bf90866e5b1cb042ea83ed2a31d"} Jul 10 22:22:39.805618 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.805593 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dwzdh" event={"ID":"7a79673b-9490-4771-a6ff-5dd650bf6c28","Type":"ContainerStarted","Data":"997ae8abeda9ae15c8dc11e8a6dadf37aa2d25474d7ff4c66c2a3c87bbd275be"} Jul 10 22:22:39.807455 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.807435 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:22:39.807747 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.807729 2565 generic.go:356] "Generic (PLEG): container finished" podID="4c95ccb2-4649-40f6-ae4b-9cdd312797ec" containerID="de7eb6ba2c5cf500a85150d41f24950946a918e539bcaee310bcdfca1391aa29" exitCode=1 Jul 10 22:22:39.807837 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.807815 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" event={"ID":"4c95ccb2-4649-40f6-ae4b-9cdd312797ec","Type":"ContainerStarted","Data":"e34fb643cd069182f9a234475287fb242ce624a15de8c8190e7a6ec9bfea187a"} Jul 10 22:22:39.807902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.807877 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" event={"ID":"4c95ccb2-4649-40f6-ae4b-9cdd312797ec","Type":"ContainerDied","Data":"de7eb6ba2c5cf500a85150d41f24950946a918e539bcaee310bcdfca1391aa29"} Jul 10 22:22:39.807954 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.807920 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" event={"ID":"4c95ccb2-4649-40f6-ae4b-9cdd312797ec","Type":"ContainerStarted","Data":"a8647bc7a6ba263271b06b0af946e17d3b8c3d0c34981ad032e4c3ce5bc43eaf"} Jul 10 22:22:39.809229 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.809210 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" event={"ID":"93168b90-d2d0-4adb-8b3f-e825f87989cc","Type":"ContainerStarted","Data":"f0bf84e167494e7e18d0b00e695fa3cdfc560da5a9bf56a91e00facebff76f96"} Jul 10 22:22:39.810567 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.810549 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5zb6h" event={"ID":"ee84dbb8-64c7-46d8-840a-94cc40156e29","Type":"ContainerStarted","Data":"a9aa41153a87ccc43ae200a7d32ff0d06c94a37656b61293b148f245d00aad18"} Jul 10 22:22:39.811989 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.811967 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-95zkp" event={"ID":"428ea365-0919-4562-bd39-f10dfdcda1d5","Type":"ContainerStarted","Data":"2d93fc015b83b164941b90eef70270196053c856b227494e1e87d08ef02dacbb"} Jul 10 22:22:39.813332 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.813315 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-st2pb" event={"ID":"3f578f81-ae6f-44cf-8e1f-746894c94c69","Type":"ContainerStarted","Data":"35bbef2348e4bc4a84b47b84c539efa91114741b589b5d31ed75987d886ce7e6"} Jul 10 22:22:39.814627 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.814606 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g96gd" event={"ID":"9f0fd957-b6e9-4ba3-9d5d-8ed753694c91","Type":"ContainerStarted","Data":"2119141736c4ad6a2da765531e7554427eddc604a170ff601bff7810e6ea500d"} Jul 10 22:22:39.816187 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.816162 2565 generic.go:356] "Generic (PLEG): container finished" podID="1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc" containerID="996edb3cc2f7207e042efe6ea44995dc7e9ad43f04c312620cff911556d725a5" exitCode=0 Jul 10 22:22:39.816279 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.816200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ln9m5" event={"ID":"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc","Type":"ContainerDied","Data":"996edb3cc2f7207e042efe6ea44995dc7e9ad43f04c312620cff911556d725a5"} Jul 10 22:22:39.837886 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.837683 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" podStartSLOduration=21.837664444 podStartE2EDuration="21.837664444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:39.8199698 +0000 UTC m=+23.652854186" watchObservedRunningTime="2026-07-10 22:22:39.837664444 +0000 UTC m=+23.670548826" Jul 10 22:22:39.838253 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.838227 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-st2pb" podStartSLOduration=3.480390691 podStartE2EDuration="22.838215245s" totalImagesPullingTime="19.357824554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:39.837629157 +0000 UTC m=+23.670513541" watchObservedRunningTime="2026-07-10 22:22:39.838215245 +0000 UTC m=+23.671099627" Jul 10 22:22:39.852762 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.852715 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5zb6h" podStartSLOduration=3.471650461 podStartE2EDuration="22.852700219s" totalImagesPullingTime="19.381049758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:39.85227203 +0000 UTC m=+23.685156416" watchObservedRunningTime="2026-07-10 22:22:39.852700219 +0000 UTC m=+23.685584602" Jul 10 22:22:39.866492 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.866452 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-95zkp" podStartSLOduration=2.5292463769999998 podStartE2EDuration="21.866437472s" totalImagesPullingTime="19.337191095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:39.866289661 +0000 UTC m=+23.699174043" watchObservedRunningTime="2026-07-10 22:22:39.866437472 +0000 UTC m=+23.699321855" Jul 10 22:22:39.887946 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.887904 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dwzdh" podStartSLOduration=2.506468394 podStartE2EDuration="21.887888746s" totalImagesPullingTime="19.381420352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:39.887606942 +0000 UTC m=+23.720491326" watchObservedRunningTime="2026-07-10 22:22:39.887888746 +0000 UTC m=+23.720773134" Jul 10 22:22:39.923008 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.922946 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g96gd" podStartSLOduration=3.5614256109999998 podStartE2EDuration="22.922930783s" totalImagesPullingTime="19.361505172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:39.922683844 +0000 UTC m=+23.755568237" watchObservedRunningTime="2026-07-10 22:22:39.922930783 +0000 UTC m=+23.755815166" Jul 10 22:22:39.935834 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:39.935774 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:40.013814 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:40.013764 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:40.110913 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:40.110787 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:40.685503 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:40.685470 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:40.686230 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:40.685583 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:40.822077 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:40.822049 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:22:40.822494 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:40.822459 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" event={"ID":"4c95ccb2-4649-40f6-ae4b-9cdd312797ec","Type":"ContainerStarted","Data":"e846367b1633b0e2a16aac984b9011629584e04760f777af1d121c9d673b75ea"} Jul 10 22:22:40.822601 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:40.822565 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" event={"ID":"4c95ccb2-4649-40f6-ae4b-9cdd312797ec","Type":"ContainerStarted","Data":"a7d5bc0377374e3b5ca13d3030a3558ade513815db821d5e5ddfca3596ab805a"} Jul 10 22:22:40.822660 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:40.822620 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" event={"ID":"4c95ccb2-4649-40f6-ae4b-9cdd312797ec","Type":"ContainerStarted","Data":"b83d6d65a3308679c2113bebc5bf1ecb8293560ccb06ae6b994ae67626c52b79"} Jul 10 22:22:40.824287 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:40.824250 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pnk69" event={"ID":"60cf880a-7e60-46a1-90bd-3539c28c5a1e","Type":"ContainerStarted","Data":"ed52254a402f2eb3e8d0d56db8693867c2d4239fcb12a230805d0e606de14449"} Jul 10 22:22:40.844959 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:40.844905 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pnk69" podStartSLOduration=4.47011686 podStartE2EDuration="23.844884556s" totalImagesPullingTime="19.374767696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:40.843907254 +0000 UTC m=+24.676791637" watchObservedRunningTime="2026-07-10 22:22:40.844884556 +0000 UTC m=+24.677768940" Jul 10 22:22:40.850283 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:40.850259 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 22:22:40.853553 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:40.853518 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:40.972363 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:40.972319 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:41.037323 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:41.037283 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:41.619915 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:41.619741 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T22:22:40.850279863Z","UUID":"5a1d8746-8dce-4a75-a4bf-7afeecd07a38","Handler":null,"Name":"","Endpoint":""} Jul 10 22:22:41.621605 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:41.621582 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 22:22:41.621765 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:41.621622 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 22:22:41.685832 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:41.685421 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:41.685832 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:41.685437 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:41.685832 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:41.685559 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:41.685832 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:41.685653 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:41.828819 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:41.828752 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" event={"ID":"93168b90-d2d0-4adb-8b3f-e825f87989cc","Type":"ContainerStarted","Data":"2daf270519c0a88cb6999b7e9f57ac92ff4f09c18c63febcb022b4251b49d86b"} Jul 10 22:22:42.276422 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:42.276341 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-95zkp" Jul 10 22:22:42.277091 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:42.277070 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-95zkp" Jul 10 22:22:42.685526 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:42.685446 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:42.685677 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:42.685581 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:42.835076 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:42.834896 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:22:42.835586 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:42.835444 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" event={"ID":"4c95ccb2-4649-40f6-ae4b-9cdd312797ec","Type":"ContainerStarted","Data":"17346b01f00acd20f81e8663c757295470d75d28fdb62165d26379f5a7035627"} Jul 10 22:22:42.837746 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:42.837717 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" event={"ID":"93168b90-d2d0-4adb-8b3f-e825f87989cc","Type":"ContainerStarted","Data":"28fd4cb04577f79a7d73ec4bfa217bf17a5045762f7466e923b16bdc11dc2b3b"} Jul 10 22:22:42.855857 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:42.855805 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r2jk" podStartSLOduration=3.189093547 podStartE2EDuration="24.855774916s" totalImagesPullingTime="21.666681369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:22:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:42.854945065 +0000 UTC m=+26.687829453" watchObservedRunningTime="2026-07-10 22:22:42.855774916 +0000 UTC m=+26.688659305" Jul 10 22:22:43.684777 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:43.684743 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:43.685032 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:43.684749 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:43.685032 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:43.684948 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:43.685150 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:43.685102 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:43.864569 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:43.864538 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-95zkp" Jul 10 22:22:43.865158 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:43.864676 2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 22:22:43.865494 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:43.865193 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-95zkp" Jul 10 22:22:43.923037 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:43.923001 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:44.501385 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:44.501287 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:44.553381 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:44.553338 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:44.684685 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:44.684656 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:44.684841 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:44.684816 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:44.760073 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:44.759979 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:44.847311 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:44.847155 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:22:44.847830 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:44.847786 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" event={"ID":"4c95ccb2-4649-40f6-ae4b-9cdd312797ec","Type":"ContainerStarted","Data":"803f2e1e57a1d4af9743299f38b02582fcd11fdeeb0f32891e5e39942933ce09"} Jul 10 22:22:44.848120 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:44.848105 2565 scope.go:117] "RemoveContainer" containerID="de7eb6ba2c5cf500a85150d41f24950946a918e539bcaee310bcdfca1391aa29" Jul 10 22:22:44.859276 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:44.859251 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:45.688776 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:45.688747 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:45.689458 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:45.688875 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:45.689458 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:45.688925 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:45.689458 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:45.689038 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:45.853071 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:45.853043 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:22:45.853454 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:45.853428 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" event={"ID":"4c95ccb2-4649-40f6-ae4b-9cdd312797ec","Type":"ContainerStarted","Data":"75b44075d2d5a373b26b0ec276fbf86e3f82ba6722d64c95546c6340bb46e160"} Jul 10 22:22:45.853649 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:45.853630 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:45.853748 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:45.853694 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:45.853748 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:45.853741 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:45.855584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:45.855555 2565 generic.go:356] "Generic (PLEG): container finished" podID="1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc" containerID="e01f0ea0be52a7d1a796cdfb21c219840c8a9821d37a2b86e0e71a1e25bd5c90" exitCode=0 Jul 10 22:22:45.855681 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:45.855599 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ln9m5" event={"ID":"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc","Type":"ContainerDied","Data":"e01f0ea0be52a7d1a796cdfb21c219840c8a9821d37a2b86e0e71a1e25bd5c90"} Jul 10 22:22:45.869387 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:45.869363 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:45.869489 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:45.869465 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:22:45.888039 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:45.887988 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" podStartSLOduration=8.489829799 podStartE2EDuration="27.887974442s" totalImagesPullingTime="19.398144643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:45.884427949 +0000 UTC m=+29.717312334" watchObservedRunningTime="2026-07-10 22:22:45.887974442 +0000 UTC m=+29.720858844" Jul 10 22:22:46.685194 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:46.685153 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:46.685434 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:46.685296 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:46.861443 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:46.861414 2565 generic.go:356] "Generic (PLEG): container finished" podID="1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc" containerID="9dc4fedaa6a5bf898d3611eb73cd4be6be770d157b69922ddc3bf84500ebc1da" exitCode=0 Jul 10 22:22:46.861896 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:46.861487 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ln9m5" event={"ID":"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc","Type":"ContainerDied","Data":"9dc4fedaa6a5bf898d3611eb73cd4be6be770d157b69922ddc3bf84500ebc1da"} Jul 10 22:22:46.962318 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:46.962221 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vzlrd"] Jul 10 22:22:46.962481 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:46.962345 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:46.962481 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:46.962460 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:46.965839 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:46.965751 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b4thn"] Jul 10 22:22:46.965974 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:46.965884 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:46.965974 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:46.965960 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:46.966394 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:46.966373 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nwdcr"] Jul 10 22:22:46.966493 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:46.966479 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:46.966607 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:46.966585 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:47.396074 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:47.395896 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:47.455455 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:47.455405 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:47.520194 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:47.520163 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:47.617680 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:47.617645 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:48.685043 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:48.684966 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:48.685461 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:48.685078 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:48.685461 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:48.685090 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:48.685461 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:48.685118 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:48.685461 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:48.685227 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:48.685461 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:48.685300 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:48.867942 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:48.867904 2565 generic.go:356] "Generic (PLEG): container finished" podID="1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc" containerID="6a843fb93cba47766f0a20eed8d0503311a93b067d5010bec947fa64100e9bde" exitCode=0 Jul 10 22:22:48.868099 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:48.867960 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ln9m5" event={"ID":"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc","Type":"ContainerDied","Data":"6a843fb93cba47766f0a20eed8d0503311a93b067d5010bec947fa64100e9bde"} Jul 10 22:22:50.685124 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:50.685087 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:50.685825 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:50.685098 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:50.685825 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:50.685087 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:50.685825 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:50.685225 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwdcr" podUID="897ef813-52ab-407d-8429-2656360c5160" Jul 10 22:22:50.685825 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:50.685417 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4thn" podUID="14b9b95f-6755-4116-bf6a-317e009798e2" Jul 10 22:22:50.685825 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:50.685440 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlrd" podUID="dc25ad9c-ea7d-40af-85e5-54b27aef019b" Jul 10 22:22:50.745825 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:50.745763 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:51.313736 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.313703 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs\") pod \"network-metrics-daemon-nwdcr\" (UID: \"897ef813-52ab-407d-8429-2656360c5160\") " pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:51.313941 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:51.313875 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:51.313985 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:51.313948 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs podName:897ef813-52ab-407d-8429-2656360c5160 nodeName:}" failed. No retries permitted until 2026-07-10 22:23:23.313928285 +0000 UTC m=+67.146812660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs") pod "network-metrics-daemon-nwdcr" (UID: "897ef813-52ab-407d-8429-2656360c5160") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:51.415051 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.415013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:51.415227 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:51.415173 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:51.415302 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:51.415252 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret podName:14b9b95f-6755-4116-bf6a-317e009798e2 nodeName:}" failed. No retries permitted until 2026-07-10 22:23:23.415228065 +0000 UTC m=+67.248112427 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret") pod "global-pull-secret-syncer-b4thn" (UID: "14b9b95f-6755-4116-bf6a-317e009798e2") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:51.515675 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.515639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwsn\" (UniqueName: \"kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn\") pod \"network-check-target-vzlrd\" (UID: \"dc25ad9c-ea7d-40af-85e5-54b27aef019b\") " pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:51.515876 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:51.515818 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:22:51.515876 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:51.515854 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:22:51.515876 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:51.515866 2565 projected.go:196] Error preparing data for projected volume kube-api-access-fvwsn for pod openshift-network-diagnostics/network-check-target-vzlrd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:51.516052 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:51.515931 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn podName:dc25ad9c-ea7d-40af-85e5-54b27aef019b nodeName:}" failed. No retries permitted until 2026-07-10 22:23:23.515910059 +0000 UTC m=+67.348794440 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvwsn" (UniqueName: "kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn") pod "network-check-target-vzlrd" (UID: "dc25ad9c-ea7d-40af-85e5-54b27aef019b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:51.809515 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.809484 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeReady" Jul 10 22:22:51.810084 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.809646 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 22:22:51.850195 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.850142 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vmg2n"] Jul 10 22:22:51.853126 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.853093 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-84fbb"] Jul 10 22:22:51.853286 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.853242 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:51.855655 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.855335 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-84fbb" Jul 10 22:22:51.855655 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.855398 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 22:22:51.855655 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.855551 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t9ksf\"" Jul 10 22:22:51.855953 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.855926 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 22:22:51.857735 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.857713 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 22:22:51.857859 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.857756 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 22:22:51.857859 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.857775 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 22:22:51.857975 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.857900 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wnjc6\"" Jul 10 22:22:51.862071 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.861687 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vmg2n"] Jul 10 22:22:51.864761 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.864709 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-84fbb"] Jul 10 22:22:51.965412 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:51.965361 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:52.019554 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.019426 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr78l\" (UniqueName: \"kubernetes.io/projected/53a16bcf-6853-40fb-99ba-a55039d01d3d-kube-api-access-pr78l\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:52.019554 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.019477 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53a16bcf-6853-40fb-99ba-a55039d01d3d-config-volume\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:52.019779 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.019557 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:52.019779 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.019618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fj2\" (UniqueName: \"kubernetes.io/projected/38a4df0c-b54e-4152-868e-f71e9f3e7e32-kube-api-access-j9fj2\") pod \"ingress-canary-84fbb\" (UID: \"38a4df0c-b54e-4152-868e-f71e9f3e7e32\") " pod="openshift-ingress-canary/ingress-canary-84fbb" Jul 10 22:22:52.019779 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.019653 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38a4df0c-b54e-4152-868e-f71e9f3e7e32-cert\") pod \"ingress-canary-84fbb\" (UID: \"38a4df0c-b54e-4152-868e-f71e9f3e7e32\") " pod="openshift-ingress-canary/ingress-canary-84fbb" Jul 10 22:22:52.019779 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.019737 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a16bcf-6853-40fb-99ba-a55039d01d3d-tmp-dir\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:52.065460 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.065421 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:52.121071 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.121031 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38a4df0c-b54e-4152-868e-f71e9f3e7e32-cert\") pod \"ingress-canary-84fbb\" (UID: \"38a4df0c-b54e-4152-868e-f71e9f3e7e32\") " pod="openshift-ingress-canary/ingress-canary-84fbb" Jul 10 22:22:52.121271 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.121109 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a16bcf-6853-40fb-99ba-a55039d01d3d-tmp-dir\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:52.121271 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.121180 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr78l\" (UniqueName: \"kubernetes.io/projected/53a16bcf-6853-40fb-99ba-a55039d01d3d-kube-api-access-pr78l\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:52.121271 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.121208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53a16bcf-6853-40fb-99ba-a55039d01d3d-config-volume\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:52.121271 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.121231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:52.121271 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.121264 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fj2\" (UniqueName: \"kubernetes.io/projected/38a4df0c-b54e-4152-868e-f71e9f3e7e32-kube-api-access-j9fj2\") pod \"ingress-canary-84fbb\" (UID: \"38a4df0c-b54e-4152-868e-f71e9f3e7e32\") " pod="openshift-ingress-canary/ingress-canary-84fbb" Jul 10 22:22:52.121546 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.121519 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a16bcf-6853-40fb-99ba-a55039d01d3d-tmp-dir\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:52.121679 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:52.121636 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 22:22:52.121746 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:52.121710 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls podName:53a16bcf-6853-40fb-99ba-a55039d01d3d nodeName:}" failed. No retries permitted until 2026-07-10 22:22:52.621688582 +0000 UTC m=+36.454572944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls") pod "dns-default-vmg2n" (UID: "53a16bcf-6853-40fb-99ba-a55039d01d3d") : secret "dns-default-metrics-tls" not found Jul 10 22:22:52.122135 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.122113 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53a16bcf-6853-40fb-99ba-a55039d01d3d-config-volume\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:52.126545 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.126522 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38a4df0c-b54e-4152-868e-f71e9f3e7e32-cert\") pod \"ingress-canary-84fbb\" (UID: \"38a4df0c-b54e-4152-868e-f71e9f3e7e32\") " pod="openshift-ingress-canary/ingress-canary-84fbb" Jul 10 22:22:52.131618 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.131566 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fj2\" (UniqueName: \"kubernetes.io/projected/38a4df0c-b54e-4152-868e-f71e9f3e7e32-kube-api-access-j9fj2\") pod \"ingress-canary-84fbb\" (UID: \"38a4df0c-b54e-4152-868e-f71e9f3e7e32\") " pod="openshift-ingress-canary/ingress-canary-84fbb" Jul 10 22:22:52.131618 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.131571 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr78l\" (UniqueName: \"kubernetes.io/projected/53a16bcf-6853-40fb-99ba-a55039d01d3d-kube-api-access-pr78l\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:52.172696 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.172661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-84fbb" Jul 10 22:22:52.626318 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.626283 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:52.626503 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:52.626454 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 22:22:52.626556 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:52.626530 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls podName:53a16bcf-6853-40fb-99ba-a55039d01d3d nodeName:}" failed. No retries permitted until 2026-07-10 22:22:53.626509246 +0000 UTC m=+37.459393610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls") pod "dns-default-vmg2n" (UID: "53a16bcf-6853-40fb-99ba-a55039d01d3d") : secret "dns-default-metrics-tls" not found Jul 10 22:22:52.685398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.685363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:22:52.685571 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.685363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:22:52.685674 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.685363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:22:52.688595 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.688571 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qbckj\"" Jul 10 22:22:52.688595 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.688578 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 22:22:52.688823 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.688580 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 22:22:52.689314 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.689296 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 22:22:52.689563 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.689545 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 22:22:52.690545 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.690519 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ts5wt\"" Jul 10 22:22:52.977453 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:52.977222 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-84fbb"] Jul 10 22:22:52.980531 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:52.980506 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a4df0c_b54e_4152_868e_f71e9f3e7e32.slice/crio-c6611997800ccd16befbdd570c95d3f091a772c36f10cc2eb0e1e2f61da651ec WatchSource:0}: Error finding container c6611997800ccd16befbdd570c95d3f091a772c36f10cc2eb0e1e2f61da651ec: Status 404 returned error can't find the container with id c6611997800ccd16befbdd570c95d3f091a772c36f10cc2eb0e1e2f61da651ec Jul 10 22:22:53.091410 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.091381 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:53.175879 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.175759 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:53.275547 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.275494 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:53.633085 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.633051 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:53.633284 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:53.633241 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 22:22:53.633350 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:53.633319 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls podName:53a16bcf-6853-40fb-99ba-a55039d01d3d nodeName:}" failed. No retries permitted until 2026-07-10 22:22:55.633298245 +0000 UTC m=+39.466182606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls") pod "dns-default-vmg2n" (UID: "53a16bcf-6853-40fb-99ba-a55039d01d3d") : secret "dns-default-metrics-tls" not found Jul 10 22:22:53.882619 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.882583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-84fbb" event={"ID":"38a4df0c-b54e-4152-868e-f71e9f3e7e32","Type":"ContainerStarted","Data":"c6611997800ccd16befbdd570c95d3f091a772c36f10cc2eb0e1e2f61da651ec"} Jul 10 22:22:53.885727 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.885653 2565 generic.go:356] "Generic (PLEG): container finished" podID="1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc" containerID="004d8c49f68ef8ae8d1ff861018f203f43c961f72dc53a738925976626a25d7d" exitCode=0 Jul 10 22:22:53.885727 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.885711 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ln9m5" event={"ID":"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc","Type":"ContainerDied","Data":"004d8c49f68ef8ae8d1ff861018f203f43c961f72dc53a738925976626a25d7d"} Jul 10 22:22:53.978327 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.978292 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dlrg8"] Jul 10 22:22:53.981144 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.981120 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:53.983650 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.983629 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 22:22:53.984063 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.984039 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 22:22:53.984148 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.984091 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 22:22:53.984206 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.984049 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 22:22:53.984432 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.984394 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bhc4w\"" Jul 10 22:22:53.994502 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:53.994477 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dlrg8"] Jul 10 22:22:54.054965 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.054931 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:54.075384 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.075300 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-wt2nr"] Jul 10 22:22:54.077281 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.077256 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-wt2nr" Jul 10 22:22:54.079781 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.079723 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 22:22:54.079950 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.079835 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bwznz\"" Jul 10 22:22:54.079950 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.079865 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 22:22:54.089150 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.089123 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-wt2nr"] Jul 10 22:22:54.138230 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.138141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.138230 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.138173 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.138230 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.138202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp6zq\" (UniqueName: \"kubernetes.io/projected/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-api-access-jp6zq\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.138479 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.138317 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-data-volume\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.138479 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.138340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-crio-socket\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.158849 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.158810 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:54.239567 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.239524 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c7wh\" (UniqueName: \"kubernetes.io/projected/24be55ba-045b-4f39-8af7-528f580ee48c-kube-api-access-7c7wh\") pod \"migrator-7f59b54f6c-wt2nr\" (UID: \"24be55ba-045b-4f39-8af7-528f580ee48c\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-wt2nr" Jul 10 22:22:54.239764 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.239623 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-data-volume\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.239764 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.239662 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-crio-socket\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.239764 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.239709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.239764 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.239738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.240266 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.239867 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6zq\" (UniqueName: \"kubernetes.io/projected/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-api-access-jp6zq\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.240266 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:54.239877 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 22:22:54.240266 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.239967 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-crio-socket\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.240266 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:54.239879 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 22:22:54.240266 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:54.240023 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm podName:e70e246e-e8b7-4709-aef7-1f23e6aecf7a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:54.740002682 +0000 UTC m=+38.572887045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dlrg8" (UID: "e70e246e-e8b7-4709-aef7-1f23e6aecf7a") : configmap "kube-rbac-proxy" not found Jul 10 22:22:54.240266 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:54.240060 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls podName:e70e246e-e8b7-4709-aef7-1f23e6aecf7a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:54.740037221 +0000 UTC m=+38.572921586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dlrg8" (UID: "e70e246e-e8b7-4709-aef7-1f23e6aecf7a") : secret "insights-runtime-extractor-tls" not found Jul 10 22:22:54.240266 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.240062 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-data-volume\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.249880 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.249810 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6zq\" (UniqueName: \"kubernetes.io/projected/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-api-access-jp6zq\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.260323 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.260289 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:54.340734 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.340691 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c7wh\" (UniqueName: \"kubernetes.io/projected/24be55ba-045b-4f39-8af7-528f580ee48c-kube-api-access-7c7wh\") pod \"migrator-7f59b54f6c-wt2nr\" (UID: \"24be55ba-045b-4f39-8af7-528f580ee48c\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-wt2nr" Jul 10 22:22:54.349507 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.349473 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c7wh\" (UniqueName: \"kubernetes.io/projected/24be55ba-045b-4f39-8af7-528f580ee48c-kube-api-access-7c7wh\") pod \"migrator-7f59b54f6c-wt2nr\" (UID: \"24be55ba-045b-4f39-8af7-528f580ee48c\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-wt2nr" Jul 10 22:22:54.387946 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.387915 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-wt2nr" Jul 10 22:22:54.520629 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.520595 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-wt2nr"] Jul 10 22:22:54.524728 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:22:54.524694 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24be55ba_045b_4f39_8af7_528f580ee48c.slice/crio-966328cfc18d3bc9075bf7459285b05b00e3f5031b223c072e6738129592bb6a WatchSource:0}: Error finding container 966328cfc18d3bc9075bf7459285b05b00e3f5031b223c072e6738129592bb6a: Status 404 returned error can't find the container with id 966328cfc18d3bc9075bf7459285b05b00e3f5031b223c072e6738129592bb6a Jul 10 22:22:54.583320 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.583286 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:54.640080 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.639997 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:54.743070 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.743039 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.743250 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.743079 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:54.743250 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:54.743187 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 22:22:54.743250 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:54.743209 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 22:22:54.743382 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:54.743269 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls podName:e70e246e-e8b7-4709-aef7-1f23e6aecf7a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:55.74324791 +0000 UTC m=+39.576132286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dlrg8" (UID: "e70e246e-e8b7-4709-aef7-1f23e6aecf7a") : secret "insights-runtime-extractor-tls" not found Jul 10 22:22:54.743382 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:54.743288 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm podName:e70e246e-e8b7-4709-aef7-1f23e6aecf7a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:55.743278926 +0000 UTC m=+39.576163287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dlrg8" (UID: "e70e246e-e8b7-4709-aef7-1f23e6aecf7a") : configmap "kube-rbac-proxy" not found Jul 10 22:22:54.859727 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.859684 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:54.896469 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.896167 2565 generic.go:356] "Generic (PLEG): container finished" podID="1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc" containerID="15cee1cd9ca4e0ac1149f0691c8f45e8ae11efd8288273a203f4cad022862e7a" exitCode=0 Jul 10 22:22:54.896469 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.896254 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ln9m5" event={"ID":"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc","Type":"ContainerDied","Data":"15cee1cd9ca4e0ac1149f0691c8f45e8ae11efd8288273a203f4cad022862e7a"} Jul 10 22:22:54.898077 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.898047 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-wt2nr" event={"ID":"24be55ba-045b-4f39-8af7-528f580ee48c","Type":"ContainerStarted","Data":"966328cfc18d3bc9075bf7459285b05b00e3f5031b223c072e6738129592bb6a"} Jul 10 22:22:54.920900 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:54.920861 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:55.652924 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:55.652835 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:55.653347 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:55.652964 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 22:22:55.653347 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:55.653034 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls podName:53a16bcf-6853-40fb-99ba-a55039d01d3d nodeName:}" failed. No retries permitted until 2026-07-10 22:22:59.653019262 +0000 UTC m=+43.485903624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls") pod "dns-default-vmg2n" (UID: "53a16bcf-6853-40fb-99ba-a55039d01d3d") : secret "dns-default-metrics-tls" not found Jul 10 22:22:55.753943 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:55.753914 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:55.754048 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:55.753950 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:55.754048 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:55.754037 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 22:22:55.754148 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:55.754049 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 22:22:55.754148 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:55.754104 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm podName:e70e246e-e8b7-4709-aef7-1f23e6aecf7a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:57.754083575 +0000 UTC m=+41.586967942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dlrg8" (UID: "e70e246e-e8b7-4709-aef7-1f23e6aecf7a") : configmap "kube-rbac-proxy" not found Jul 10 22:22:55.754148 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:55.754125 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls podName:e70e246e-e8b7-4709-aef7-1f23e6aecf7a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:57.754115127 +0000 UTC m=+41.586999496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dlrg8" (UID: "e70e246e-e8b7-4709-aef7-1f23e6aecf7a") : secret "insights-runtime-extractor-tls" not found Jul 10 22:22:55.901923 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:55.901888 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-84fbb" event={"ID":"38a4df0c-b54e-4152-868e-f71e9f3e7e32","Type":"ContainerStarted","Data":"5923c245dc278b6cd814843c4fe42fa585e8cae14450ff6e3876794d52d6f2b2"} Jul 10 22:22:55.905898 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:55.905818 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ln9m5" event={"ID":"1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc","Type":"ContainerStarted","Data":"8564138e6822a66672e37b598dcf9406a32935511089e907bc4ce708e5796fce"} Jul 10 22:22:55.917567 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:55.917513 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-84fbb" podStartSLOduration=2.17419035 podStartE2EDuration="4.917497453s" totalImagesPullingTime="2.743307103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:55.916079831 +0000 UTC m=+39.748964213" watchObservedRunningTime="2026-07-10 22:22:55.917497453 +0000 UTC m=+39.750381835" Jul 10 22:22:55.936487 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:55.936435 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ln9m5" podStartSLOduration=13.455562075 podStartE2EDuration="38.936418088s" totalImagesPullingTime="25.480856013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:22:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:55.935239392 +0000 UTC m=+39.768123776" watchObservedRunningTime="2026-07-10 22:22:55.936418088 +0000 UTC m=+39.769302470" Jul 10 22:22:55.974436 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:55.974411 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g96gd_9f0fd957-b6e9-4ba3-9d5d-8ed753694c91/dns-node-resolver/0.log" Jul 10 22:22:56.138642 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:56.138602 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:56.191917 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:56.191843 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:56.482617 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:56.482577 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:56.540181 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:56.540149 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:56.775566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:56.775490 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5zb6h_ee84dbb8-64c7-46d8-840a-94cc40156e29/node-ca/0.log" Jul 10 22:22:57.487195 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:57.487165 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:57.770122 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:57.770038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:57.770122 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:57.770075 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:22:57.770320 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:57.770183 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 22:22:57.770320 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:57.770184 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 22:22:57.770320 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:57.770256 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls podName:e70e246e-e8b7-4709-aef7-1f23e6aecf7a nodeName:}" failed. No retries permitted until 2026-07-10 22:23:01.770234371 +0000 UTC m=+45.603118747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dlrg8" (UID: "e70e246e-e8b7-4709-aef7-1f23e6aecf7a") : secret "insights-runtime-extractor-tls" not found Jul 10 22:22:57.770320 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:57.770276 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm podName:e70e246e-e8b7-4709-aef7-1f23e6aecf7a nodeName:}" failed. No retries permitted until 2026-07-10 22:23:01.770268812 +0000 UTC m=+45.603153172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dlrg8" (UID: "e70e246e-e8b7-4709-aef7-1f23e6aecf7a") : configmap "kube-rbac-proxy" not found Jul 10 22:22:57.776181 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:57.776157 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-84fbb_38a4df0c-b54e-4152-868e-f71e9f3e7e32/serve-healthcheck-canary/0.log" Jul 10 22:22:57.911655 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:57.911622 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-wt2nr" event={"ID":"24be55ba-045b-4f39-8af7-528f580ee48c","Type":"ContainerStarted","Data":"bb3bda3f34ccee804839b2e7f290ec52c52c4609e8847e158d73e286e8f4c181"} Jul 10 22:22:57.911833 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:57.911682 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-wt2nr" event={"ID":"24be55ba-045b-4f39-8af7-528f580ee48c","Type":"ContainerStarted","Data":"ff169af9bff1c0c5eb22e6403d535ed9e6bc7ba23d38978fa9e773d300ca9e12"} Jul 10 22:22:57.927583 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:57.927536 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-wt2nr" podStartSLOduration=1.5548554 podStartE2EDuration="3.927520591s" totalImagesPullingTime="2.372665191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:57.926859104 +0000 UTC m=+41.759743489" watchObservedRunningTime="2026-07-10 22:22:57.927520591 +0000 UTC m=+41.760404973" Jul 10 22:22:58.317199 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:58.317160 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:58.435218 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:58.435187 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:58.535352 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:58.535320 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:58.635528 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:58.635453 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:58.688730 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:58.688684 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:59.219422 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:59.219392 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:59.319458 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:59.319425 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:59.420610 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:59.420574 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:59.686899 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:22:59.686869 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:22:59.687072 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:59.686973 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 22:22:59.687072 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:22:59.687029 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls podName:53a16bcf-6853-40fb-99ba-a55039d01d3d nodeName:}" failed. No retries permitted until 2026-07-10 22:23:07.687015541 +0000 UTC m=+51.519899902 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls") pod "dns-default-vmg2n" (UID: "53a16bcf-6853-40fb-99ba-a55039d01d3d") : secret "dns-default-metrics-tls" not found Jul 10 22:23:00.219521 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:00.219485 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:00.279759 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:00.279726 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:00.614559 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:00.614525 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:01.021772 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:01.021727 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:01.215874 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:01.215840 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:01.617975 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:01.617938 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:01.806291 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:01.806256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:23:01.806439 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:01.806345 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:23:01.806439 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:23:01.806383 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 22:23:01.806439 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:23:01.806389 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 22:23:01.806548 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:23:01.806447 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm podName:e70e246e-e8b7-4709-aef7-1f23e6aecf7a nodeName:}" failed. No retries permitted until 2026-07-10 22:23:09.806432963 +0000 UTC m=+53.639317367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dlrg8" (UID: "e70e246e-e8b7-4709-aef7-1f23e6aecf7a") : configmap "kube-rbac-proxy" not found Jul 10 22:23:01.806548 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:23:01.806476 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls podName:e70e246e-e8b7-4709-aef7-1f23e6aecf7a nodeName:}" failed. No retries permitted until 2026-07-10 22:23:09.80646156 +0000 UTC m=+53.639345922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dlrg8" (UID: "e70e246e-e8b7-4709-aef7-1f23e6aecf7a") : secret "insights-runtime-extractor-tls" not found Jul 10 22:23:02.014843 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:02.014812 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:02.246906 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:02.246867 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:02.421552 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:02.421471 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:02.624996 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:02.624957 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:03.023129 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:03.023093 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:03.123256 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:03.123225 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:03.223523 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:03.223489 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:03.621987 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:03.621956 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:03.672198 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:03.672167 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:04.418005 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:04.417968 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:04.610095 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:04.610058 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:04.664209 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:04.664170 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:05.013080 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:05.013049 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:05.210363 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:05.210331 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:05.619360 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:05.619324 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:06.221356 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:06.221327 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:06.277655 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:06.277621 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:06.644697 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:06.644621 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:07.017094 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:07.017063 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:07.413418 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:07.413337 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:07.619242 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:07.619213 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:07.748398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:07.748370 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:23:07.751591 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:07.751572 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53a16bcf-6853-40fb-99ba-a55039d01d3d-metrics-tls\") pod \"dns-default-vmg2n\" (UID: \"53a16bcf-6853-40fb-99ba-a55039d01d3d\") " pod="openshift-dns/dns-default-vmg2n" Jul 10 22:23:07.777261 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:07.767920 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vmg2n" Jul 10 22:23:07.905627 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:07.905459 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vmg2n"] Jul 10 22:23:07.908083 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:23:07.908056 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a16bcf_6853_40fb_99ba_a55039d01d3d.slice/crio-db45b103c9b4ebfabde03f3eb77458ba39dbcc6aec174bd2f745ae20dff0e0f6 WatchSource:0}: Error finding container db45b103c9b4ebfabde03f3eb77458ba39dbcc6aec174bd2f745ae20dff0e0f6: Status 404 returned error can't find the container with id db45b103c9b4ebfabde03f3eb77458ba39dbcc6aec174bd2f745ae20dff0e0f6 Jul 10 22:23:07.942557 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:07.942522 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vmg2n" event={"ID":"53a16bcf-6853-40fb-99ba-a55039d01d3d","Type":"ContainerStarted","Data":"db45b103c9b4ebfabde03f3eb77458ba39dbcc6aec174bd2f745ae20dff0e0f6"} Jul 10 22:23:08.213026 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:08.212939 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:08.617163 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:08.617114 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:08.816777 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:08.816738 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:09.013941 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:09.013901 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:09.865140 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:09.865103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:23:09.865140 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:09.865142 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:23:09.865741 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:09.865720 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:23:09.867263 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:09.867242 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e70e246e-e8b7-4709-aef7-1f23e6aecf7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlrg8\" (UID: \"e70e246e-e8b7-4709-aef7-1f23e6aecf7a\") " pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:23:09.892244 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:09.892222 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dlrg8" Jul 10 22:23:10.027295 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:10.027269 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dlrg8"] Jul 10 22:23:10.029955 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:23:10.029927 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70e246e_e8b7_4709_aef7_1f23e6aecf7a.slice/crio-d22c79b2ea794208d89a253de8a7d0e4579669ccdfbab57b762f2ea40f371a7f WatchSource:0}: Error finding container d22c79b2ea794208d89a253de8a7d0e4579669ccdfbab57b762f2ea40f371a7f: Status 404 returned error can't find the container with id d22c79b2ea794208d89a253de8a7d0e4579669ccdfbab57b762f2ea40f371a7f Jul 10 22:23:10.954723 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:10.954637 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vmg2n" event={"ID":"53a16bcf-6853-40fb-99ba-a55039d01d3d","Type":"ContainerStarted","Data":"5d8c35e5e8a81a3ab7273bbec574bb198ecc322e004ebb8252b933ec8c9170be"} Jul 10 22:23:10.954723 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:10.954710 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vmg2n" event={"ID":"53a16bcf-6853-40fb-99ba-a55039d01d3d","Type":"ContainerStarted","Data":"7d0e3273156f1b49e13317118cb5c81da7cb2818f62111ab4a9d51866470ef57"} Jul 10 22:23:10.955307 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:10.954822 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vmg2n" Jul 10 22:23:10.956256 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:10.956168 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlrg8" event={"ID":"e70e246e-e8b7-4709-aef7-1f23e6aecf7a","Type":"ContainerStarted","Data":"d22c79b2ea794208d89a253de8a7d0e4579669ccdfbab57b762f2ea40f371a7f"} Jul 10 22:23:10.971925 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:10.971870 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vmg2n" podStartSLOduration=17.637849424 podStartE2EDuration="19.971849348s" totalImagesPullingTime="2.333999924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:10.970711796 +0000 UTC m=+54.803596178" watchObservedRunningTime="2026-07-10 22:23:10.971849348 +0000 UTC m=+54.804733729" Jul 10 22:23:13.967531 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:13.967437 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlrg8" event={"ID":"e70e246e-e8b7-4709-aef7-1f23e6aecf7a","Type":"ContainerStarted","Data":"65d4239951ebf70aa13564e356ea2b2fad2882662d8a77f3b773bc3c09696840"} Jul 10 22:23:14.973610 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:14.973573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlrg8" event={"ID":"e70e246e-e8b7-4709-aef7-1f23e6aecf7a","Type":"ContainerStarted","Data":"64aa2df498293c38dd29ca2dd80afc722fff167774cdb0f25c45993f441f3fd2"} Jul 10 22:23:16.462930 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.462892 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr"] Jul 10 22:23:16.502703 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.502661 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8"] Jul 10 22:23:16.502901 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.502766 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr" Jul 10 22:23:16.506372 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.506339 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 22:23:16.506505 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.506471 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 22:23:16.506744 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.506720 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 10 22:23:16.507337 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.507312 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xhcbj\"" Jul 10 22:23:16.507466 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.507316 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 22:23:16.516976 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.516949 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd9a46c3-a311-4b61-8966-efbd7a5e826d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr\" (UID: \"dd9a46c3-a311-4b61-8966-efbd7a5e826d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr" Jul 10 22:23:16.517079 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.516989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxrx9\" (UniqueName: \"kubernetes.io/projected/dd9a46c3-a311-4b61-8966-efbd7a5e826d-kube-api-access-sxrx9\") pod \"managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr\" (UID: \"dd9a46c3-a311-4b61-8966-efbd7a5e826d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr" Jul 10 22:23:16.536336 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.536301 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr"] Jul 10 22:23:16.536336 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.536340 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8"] Jul 10 22:23:16.536543 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.536425 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.539839 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.539813 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 22:23:16.540284 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.540263 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 22:23:16.540912 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.540877 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 22:23:16.540912 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.540884 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 22:23:16.578818 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.578767 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:16.618199 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.618168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd9a46c3-a311-4b61-8966-efbd7a5e826d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr\" (UID: \"dd9a46c3-a311-4b61-8966-efbd7a5e826d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr" Jul 10 22:23:16.618199 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.618201 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxrx9\" (UniqueName: \"kubernetes.io/projected/dd9a46c3-a311-4b61-8966-efbd7a5e826d-kube-api-access-sxrx9\") pod \"managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr\" (UID: \"dd9a46c3-a311-4b61-8966-efbd7a5e826d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr" Jul 10 22:23:16.618424 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.618234 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0e0aecdf-d248-4191-9b76-cb69202f3251-hub\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.618424 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.618259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e0aecdf-d248-4191-9b76-cb69202f3251-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.618529 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.618439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0e0aecdf-d248-4191-9b76-cb69202f3251-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.618529 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.618480 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0e0aecdf-d248-4191-9b76-cb69202f3251-ca\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.618529 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.618506 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0aecdf-d248-4191-9b76-cb69202f3251-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.618650 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.618549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvsg\" (UniqueName: \"kubernetes.io/projected/0e0aecdf-d248-4191-9b76-cb69202f3251-kube-api-access-5qvsg\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.620774 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.620747 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd9a46c3-a311-4b61-8966-efbd7a5e826d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr\" (UID: \"dd9a46c3-a311-4b61-8966-efbd7a5e826d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr" Jul 10 22:23:16.629344 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.629320 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxrx9\" (UniqueName: \"kubernetes.io/projected/dd9a46c3-a311-4b61-8966-efbd7a5e826d-kube-api-access-sxrx9\") pod \"managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr\" (UID: \"dd9a46c3-a311-4b61-8966-efbd7a5e826d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr" Jul 10 22:23:16.688392 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.688353 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:16.718945 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.718873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0e0aecdf-d248-4191-9b76-cb69202f3251-hub\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.718945 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.718907 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e0aecdf-d248-4191-9b76-cb69202f3251-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.719146 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.718957 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0e0aecdf-d248-4191-9b76-cb69202f3251-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.719146 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.718978 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0e0aecdf-d248-4191-9b76-cb69202f3251-ca\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.719146 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.719001 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0aecdf-d248-4191-9b76-cb69202f3251-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.719146 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.719025 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvsg\" (UniqueName: \"kubernetes.io/projected/0e0aecdf-d248-4191-9b76-cb69202f3251-kube-api-access-5qvsg\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.719903 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.719862 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0aecdf-d248-4191-9b76-cb69202f3251-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.721436 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.721416 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0e0aecdf-d248-4191-9b76-cb69202f3251-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.721540 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.721459 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0e0aecdf-d248-4191-9b76-cb69202f3251-ca\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.721721 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.721698 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0e0aecdf-d248-4191-9b76-cb69202f3251-hub\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.721769 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.721741 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e0aecdf-d248-4191-9b76-cb69202f3251-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.731830 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.731787 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvsg\" (UniqueName: \"kubernetes.io/projected/0e0aecdf-d248-4191-9b76-cb69202f3251-kube-api-access-5qvsg\") pod \"cluster-proxy-proxy-agent-67487fc995-6lbw8\" (UID: \"0e0aecdf-d248-4191-9b76-cb69202f3251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.786178 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.786143 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:16.822213 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.822182 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr" Jul 10 22:23:16.848499 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.848476 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" Jul 10 22:23:16.958502 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:16.958452 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr"] Jul 10 22:23:17.000964 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:17.000937 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8"] Jul 10 22:23:17.076704 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:23:17.076667 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd9a46c3_a311_4b61_8966_efbd7a5e826d.slice/crio-6812b9c77db6ca43c6a0c83ee9e7458c97545aa654adbd98cc9fd3059da5b401 WatchSource:0}: Error finding container 6812b9c77db6ca43c6a0c83ee9e7458c97545aa654adbd98cc9fd3059da5b401: Status 404 returned error can't find the container with id 6812b9c77db6ca43c6a0c83ee9e7458c97545aa654adbd98cc9fd3059da5b401 Jul 10 22:23:17.077490 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:23:17.077343 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e0aecdf_d248_4191_9b76_cb69202f3251.slice/crio-4d8a293c8b87c475a6ab3d469a7b562f9d4f3d72f5dbca34fd96a2c8621ab87f WatchSource:0}: Error finding container 4d8a293c8b87c475a6ab3d469a7b562f9d4f3d72f5dbca34fd96a2c8621ab87f: Status 404 returned error can't find the container with id 4d8a293c8b87c475a6ab3d469a7b562f9d4f3d72f5dbca34fd96a2c8621ab87f Jul 10 22:23:17.883346 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:17.883311 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:17.884269 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:17.884246 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bwjng" Jul 10 22:23:17.983550 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:17.983510 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:17.987418 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:17.987387 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlrg8" event={"ID":"e70e246e-e8b7-4709-aef7-1f23e6aecf7a","Type":"ContainerStarted","Data":"3b38fba56fa2b896e3916a85828dae230a3f439561cdb862c69a731920e01dac"} Jul 10 22:23:17.988820 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:17.988762 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" event={"ID":"0e0aecdf-d248-4191-9b76-cb69202f3251","Type":"ContainerStarted","Data":"4d8a293c8b87c475a6ab3d469a7b562f9d4f3d72f5dbca34fd96a2c8621ab87f"} Jul 10 22:23:17.990209 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:17.990186 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr" event={"ID":"dd9a46c3-a311-4b61-8966-efbd7a5e826d","Type":"ContainerStarted","Data":"6812b9c77db6ca43c6a0c83ee9e7458c97545aa654adbd98cc9fd3059da5b401"} Jul 10 22:23:18.004968 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:18.004919 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dlrg8" podStartSLOduration=18.050437711 podStartE2EDuration="25.004902491s" totalImagesPullingTime="6.95446478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:22:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:18.00443 +0000 UTC m=+61.837314384" watchObservedRunningTime="2026-07-10 22:23:18.004902491 +0000 UTC m=+61.837786876" Jul 10 22:23:18.083525 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:18.083493 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:18.140305 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:18.140204 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:19.280218 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:19.280183 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:19.332235 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:19.332199 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:20.000231 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:20.000195 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr" event={"ID":"dd9a46c3-a311-4b61-8966-efbd7a5e826d","Type":"ContainerStarted","Data":"0de614af10023a1530880ed24b371f52c12d5f0b14ee1e985d213e6f44cefb22"} Jul 10 22:23:20.016418 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:20.016271 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c7f88cf76-8jkqr" podStartSLOduration=1.207963575 podStartE2EDuration="4.016248025s" totalImagesPullingTime="2.80828445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:20.015911924 +0000 UTC m=+63.848796308" watchObservedRunningTime="2026-07-10 22:23:20.016248025 +0000 UTC m=+63.849132410" Jul 10 22:23:20.962566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:20.962529 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vmg2n" Jul 10 22:23:22.009121 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:22.009083 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" event={"ID":"0e0aecdf-d248-4191-9b76-cb69202f3251","Type":"ContainerStarted","Data":"4074372a91d4a05d81d73c45a159e2fa678675473665997ac2263619a09fdb06"} Jul 10 22:23:22.009121 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:22.009127 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" event={"ID":"0e0aecdf-d248-4191-9b76-cb69202f3251","Type":"ContainerStarted","Data":"5f0cd90d0e3877b308ea8c1498c9f586a66606d01c36b204f684456f92ca3990"} Jul 10 22:23:22.009644 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:22.009139 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" event={"ID":"0e0aecdf-d248-4191-9b76-cb69202f3251","Type":"ContainerStarted","Data":"c957a9c68604e94b1088638108e83c5fa0d05f3c0b6de79908eb5a2c244af45e"} Jul 10 22:23:22.032400 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:22.032353 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67487fc995-6lbw8" podStartSLOduration=1.99271784 podStartE2EDuration="6.032339238s" totalImagesPullingTime="4.039621398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:22.032204816 +0000 UTC m=+65.865089203" watchObservedRunningTime="2026-07-10 22:23:22.032339238 +0000 UTC m=+65.865223620" Jul 10 22:23:22.890154 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:22.890121 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:22.991478 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:22.991446 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:23.165351 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.165263 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:23.261622 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.261594 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:23.336192 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.336164 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:23.370460 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.370425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs\") pod \"network-metrics-daemon-nwdcr\" (UID: \"897ef813-52ab-407d-8429-2656360c5160\") " pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:23:23.372934 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.372914 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 22:23:23.383264 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.383235 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ef813-52ab-407d-8429-2656360c5160-metrics-certs\") pod \"network-metrics-daemon-nwdcr\" (UID: \"897ef813-52ab-407d-8429-2656360c5160\") " pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:23:23.416041 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.415975 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:23.471248 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.471206 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:23:23.473689 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.473671 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 22:23:23.483706 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.483676 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14b9b95f-6755-4116-bf6a-317e009798e2-original-pull-secret\") pod \"global-pull-secret-syncer-b4thn\" (UID: \"14b9b95f-6755-4116-bf6a-317e009798e2\") " pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:23:23.571819 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.571760 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwsn\" (UniqueName: \"kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn\") pod \"network-check-target-vzlrd\" (UID: \"dc25ad9c-ea7d-40af-85e5-54b27aef019b\") " pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:23:23.574048 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.574030 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 22:23:23.584768 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.584743 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 22:23:23.596225 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.596199 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvwsn\" (UniqueName: \"kubernetes.io/projected/dc25ad9c-ea7d-40af-85e5-54b27aef019b-kube-api-access-fvwsn\") pod \"network-check-target-vzlrd\" (UID: \"dc25ad9c-ea7d-40af-85e5-54b27aef019b\") " pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:23:23.600838 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.600817 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qbckj\"" Jul 10 22:23:23.607340 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.607322 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ts5wt\"" Jul 10 22:23:23.609021 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.609006 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwdcr" Jul 10 22:23:23.612259 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.612243 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4thn" Jul 10 22:23:23.615591 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.615573 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:23:23.759780 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.759747 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nwdcr"] Jul 10 22:23:23.763352 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:23:23.763319 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897ef813_52ab_407d_8429_2656360c5160.slice/crio-fedc3f1b63188e4d74ac2b0b9a48b0332b3b42d41a8ea087688aeea0d21953b0 WatchSource:0}: Error finding container fedc3f1b63188e4d74ac2b0b9a48b0332b3b42d41a8ea087688aeea0d21953b0: Status 404 returned error can't find the container with id fedc3f1b63188e4d74ac2b0b9a48b0332b3b42d41a8ea087688aeea0d21953b0 Jul 10 22:23:23.769009 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.768981 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vzlrd"] Jul 10 22:23:23.772381 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:23:23.772353 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc25ad9c_ea7d_40af_85e5_54b27aef019b.slice/crio-eec5b2b341e2d1a6e0e1f6938d96a215437d25e5286717e5e21e01716944b09b WatchSource:0}: Error finding container eec5b2b341e2d1a6e0e1f6938d96a215437d25e5286717e5e21e01716944b09b: Status 404 returned error can't find the container with id eec5b2b341e2d1a6e0e1f6938d96a215437d25e5286717e5e21e01716944b09b Jul 10 22:23:23.788245 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:23.788219 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b4thn"] Jul 10 22:23:23.791277 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:23:23.791256 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b9b95f_6755_4116_bf6a_317e009798e2.slice/crio-310a2cf58abce50a191da0c9e9a424a875f189a1f5e28cc4b407adf41b6fc641 WatchSource:0}: Error finding container 310a2cf58abce50a191da0c9e9a424a875f189a1f5e28cc4b407adf41b6fc641: Status 404 returned error can't find the container with id 310a2cf58abce50a191da0c9e9a424a875f189a1f5e28cc4b407adf41b6fc641 Jul 10 22:23:24.016307 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:24.016278 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vzlrd" event={"ID":"dc25ad9c-ea7d-40af-85e5-54b27aef019b","Type":"ContainerStarted","Data":"eec5b2b341e2d1a6e0e1f6938d96a215437d25e5286717e5e21e01716944b09b"} Jul 10 22:23:24.017431 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:24.017396 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b4thn" event={"ID":"14b9b95f-6755-4116-bf6a-317e009798e2","Type":"ContainerStarted","Data":"310a2cf58abce50a191da0c9e9a424a875f189a1f5e28cc4b407adf41b6fc641"} Jul 10 22:23:24.018469 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:24.018446 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwdcr" event={"ID":"897ef813-52ab-407d-8429-2656360c5160","Type":"ContainerStarted","Data":"fedc3f1b63188e4d74ac2b0b9a48b0332b3b42d41a8ea087688aeea0d21953b0"} Jul 10 22:23:26.029134 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:26.029088 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwdcr" event={"ID":"897ef813-52ab-407d-8429-2656360c5160","Type":"ContainerStarted","Data":"c168f3c770778a786c08599c4e5a3e58e95ed534bdb2b01644bd346012b9e541"} Jul 10 22:23:26.029610 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:26.029157 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwdcr" event={"ID":"897ef813-52ab-407d-8429-2656360c5160","Type":"ContainerStarted","Data":"9218be4ee8a7a885229e360653edcd8b813a725d8f571ec54b0cdf6310128add"} Jul 10 22:23:26.045635 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:26.045577 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nwdcr" podStartSLOduration=67.497556851 podStartE2EDuration="1m9.045555224s" totalImagesPullingTime="1.547998373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:26.045223174 +0000 UTC m=+69.878107557" watchObservedRunningTime="2026-07-10 22:23:26.045555224 +0000 UTC m=+69.878439611" Jul 10 22:23:26.129187 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:26.129151 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:26.229329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:26.229289 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:26.352558 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:26.352467 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:26.421888 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:26.421786 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:26.575913 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:26.575869 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:28.372752 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:28.372720 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:28.776325 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:28.776290 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:28.835855 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:28.835821 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:29.041821 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:29.041708 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vzlrd" event={"ID":"dc25ad9c-ea7d-40af-85e5-54b27aef019b","Type":"ContainerStarted","Data":"08c7c3279ddf0399a471aceb1e8e9e156ff7c65fdcd211279e56f8d0db67681d"} Jul 10 22:23:29.041994 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:29.041819 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:23:29.043196 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:29.043173 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b4thn" event={"ID":"14b9b95f-6755-4116-bf6a-317e009798e2","Type":"ContainerStarted","Data":"e199e19cc6192fd7d55a292b81727d771bfab5438eabeec4f2302daff8192276"} Jul 10 22:23:29.057304 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:29.057259 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vzlrd" podStartSLOduration=66.416783278 podStartE2EDuration="1m11.057243821s" totalImagesPullingTime="4.640460543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:29.056026992 +0000 UTC m=+72.888911385" watchObservedRunningTime="2026-07-10 22:23:29.057243821 +0000 UTC m=+72.890128252" Jul 10 22:23:29.069477 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:29.069430 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b4thn" podStartSLOduration=66.442873284 podStartE2EDuration="1m11.069414809s" totalImagesPullingTime="4.626541525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:29.069020471 +0000 UTC m=+72.901904853" watchObservedRunningTime="2026-07-10 22:23:29.069414809 +0000 UTC m=+72.902299192" Jul 10 22:23:30.087173 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.087138 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:30.481058 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.481026 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:30.734519 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.734439 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:30.800601 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.800568 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-68wm9"] Jul 10 22:23:30.802994 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.802972 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:30.805273 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.805249 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 22:23:30.805396 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.805251 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 22:23:30.805453 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.805424 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 22:23:30.806671 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.806643 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jcm6v\"" Jul 10 22:23:30.806781 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.806721 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 22:23:30.806781 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.806773 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 22:23:30.807054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.807032 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 22:23:30.839731 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.839696 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:30.930094 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.930066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-accelerators-collector-config\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:30.930266 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.930098 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc95ce3a-45f8-4017-bd90-510e022d2feb-metrics-client-ca\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:30.930266 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.930123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc95ce3a-45f8-4017-bd90-510e022d2feb-root\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:30.930266 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.930143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-tls\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:30.930266 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.930183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-wtmp\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:30.930266 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.930204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767pw\" (UniqueName: \"kubernetes.io/projected/bc95ce3a-45f8-4017-bd90-510e022d2feb-kube-api-access-767pw\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:30.930266 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.930222 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:30.930458 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.930281 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-textfile\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:30.930458 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.930307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc95ce3a-45f8-4017-bd90-510e022d2feb-sys\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:30.933363 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:30.933335 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:31.030461 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.030368 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:31.030630 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.030541 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-wtmp\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.030630 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.030567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767pw\" (UniqueName: \"kubernetes.io/projected/bc95ce3a-45f8-4017-bd90-510e022d2feb-kube-api-access-767pw\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.030630 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.030591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.030630 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.030611 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-textfile\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.030630 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.030626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc95ce3a-45f8-4017-bd90-510e022d2feb-sys\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.030897 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.030687 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc95ce3a-45f8-4017-bd90-510e022d2feb-sys\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.030897 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.030726 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-accelerators-collector-config\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.030897 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.030755 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-wtmp\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.030897 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.030780 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc95ce3a-45f8-4017-bd90-510e022d2feb-metrics-client-ca\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.030897 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.030828 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc95ce3a-45f8-4017-bd90-510e022d2feb-root\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.030897 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.030852 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-tls\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.031138 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.030944 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc95ce3a-45f8-4017-bd90-510e022d2feb-root\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.031138 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:23:31.030981 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 22:23:31.031138 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.031010 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-textfile\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.031138 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:23:31.031047 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-tls podName:bc95ce3a-45f8-4017-bd90-510e022d2feb nodeName:}" failed. No retries permitted until 2026-07-10 22:23:31.531027734 +0000 UTC m=+75.363912095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-tls") pod "node-exporter-68wm9" (UID: "bc95ce3a-45f8-4017-bd90-510e022d2feb") : secret "node-exporter-tls" not found Jul 10 22:23:31.031402 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.031381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc95ce3a-45f8-4017-bd90-510e022d2feb-metrics-client-ca\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.031437 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.031381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-accelerators-collector-config\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.033009 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.032985 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.040294 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.040274 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-767pw\" (UniqueName: \"kubernetes.io/projected/bc95ce3a-45f8-4017-bd90-510e022d2feb-kube-api-access-767pw\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.130339 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.130293 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:31.255257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.255217 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:31.355635 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.355539 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:31.534443 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.534407 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-tls\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.536761 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.536735 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc95ce3a-45f8-4017-bd90-510e022d2feb-node-exporter-tls\") pod \"node-exporter-68wm9\" (UID: \"bc95ce3a-45f8-4017-bd90-510e022d2feb\") " pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.709762 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.709670 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:31.715609 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.715582 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-68wm9" Jul 10 22:23:31.725968 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:23:31.725937 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc95ce3a_45f8_4017_bd90_510e022d2feb.slice/crio-584c60fa570d097190fff48663d4ade0cba9cd996f1b1eefb389a373ede782e4 WatchSource:0}: Error finding container 584c60fa570d097190fff48663d4ade0cba9cd996f1b1eefb389a373ede782e4: Status 404 returned error can't find the container with id 584c60fa570d097190fff48663d4ade0cba9cd996f1b1eefb389a373ede782e4 Jul 10 22:23:31.775167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:31.775135 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:32.004749 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:32.004714 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:32.052833 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:32.052786 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-68wm9" event={"ID":"bc95ce3a-45f8-4017-bd90-510e022d2feb","Type":"ContainerStarted","Data":"584c60fa570d097190fff48663d4ade0cba9cd996f1b1eefb389a373ede782e4"} Jul 10 22:23:32.069772 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:32.069746 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:32.280217 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:32.280145 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:32.475968 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:32.475938 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:32.571812 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:32.571706 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:33.057538 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:33.057505 2565 generic.go:356] "Generic (PLEG): container finished" podID="bc95ce3a-45f8-4017-bd90-510e022d2feb" containerID="f906943c0f90095e447bbb72aae0f0778cb371778a8b2d1daadf7efa667cfe29" exitCode=0 Jul 10 22:23:33.057709 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:33.057573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-68wm9" event={"ID":"bc95ce3a-45f8-4017-bd90-510e022d2feb","Type":"ContainerDied","Data":"f906943c0f90095e447bbb72aae0f0778cb371778a8b2d1daadf7efa667cfe29"} Jul 10 22:23:33.281938 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:33.281902 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:33.384771 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:33.384685 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:33.482315 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:33.482282 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:33.585269 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:33.585241 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:33.683707 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:33.683625 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:33.787640 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:33.787604 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:33.881073 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:33.881039 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:33.954742 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:33.954663 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:34.063339 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:34.063301 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-68wm9" event={"ID":"bc95ce3a-45f8-4017-bd90-510e022d2feb","Type":"ContainerStarted","Data":"1687b396d622c705612f56ac62b3ca8ed863fff7608c57aa95dd849f9c57e422"} Jul 10 22:23:34.063533 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:34.063386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-68wm9" event={"ID":"bc95ce3a-45f8-4017-bd90-510e022d2feb","Type":"ContainerStarted","Data":"539aaf02333c15f328a08acc104813dbfd7bb5890a46e7d3e7ebea4b789309cb"} Jul 10 22:23:34.082764 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:34.082705 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-68wm9" podStartSLOduration=3.126023599 podStartE2EDuration="4.082689277s" totalImagesPullingTime="891.667209ms" totalInitContainerRuntime="64.998469ms" isStatefulPod=true podCreationTimestamp="2026-07-10 22:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:34.081116678 +0000 UTC m=+77.914001061" watchObservedRunningTime="2026-07-10 22:23:34.082689277 +0000 UTC m=+77.915573659" Jul 10 22:23:34.085121 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:34.085095 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:34.167327 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:34.167292 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:34.268706 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:34.268659 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:35.544708 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:35.544674 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:35.658596 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:35.658556 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:35.759321 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:35.759282 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:35.962260 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:35.962183 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:36.980957 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:36.980923 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:37.172577 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.172549 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:37.286140 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.286058 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:37.695062 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.694979 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68ffbf4cc4-cscnw"] Jul 10 22:23:37.697097 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.697079 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.699517 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.699494 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 22:23:37.699644 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.699494 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 22:23:37.700125 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.700110 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-f5j8x\"" Jul 10 22:23:37.700248 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.700111 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 22:23:37.700679 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.700665 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 22:23:37.700769 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.700706 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 22:23:37.700769 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.700714 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 22:23:37.700769 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.700706 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 22:23:37.708328 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.708308 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68ffbf4cc4-cscnw"] Jul 10 22:23:37.781875 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.781843 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/896f2237-4122-4df3-bd94-d33c09a3923a-console-serving-cert\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.781875 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.781876 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-oauth-serving-cert\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.782095 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.781920 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/896f2237-4122-4df3-bd94-d33c09a3923a-console-oauth-config\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.782095 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.781963 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-console-config\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.782095 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.781980 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-service-ca\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.782095 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.782010 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mgb8\" (UniqueName: \"kubernetes.io/projected/896f2237-4122-4df3-bd94-d33c09a3923a-kube-api-access-4mgb8\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.859571 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.859537 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:37.882363 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.882331 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/896f2237-4122-4df3-bd94-d33c09a3923a-console-oauth-config\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.882363 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.882368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-console-config\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.882591 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.882386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-service-ca\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.882591 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.882400 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgb8\" (UniqueName: \"kubernetes.io/projected/896f2237-4122-4df3-bd94-d33c09a3923a-kube-api-access-4mgb8\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.882591 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.882425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/896f2237-4122-4df3-bd94-d33c09a3923a-console-serving-cert\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.882591 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.882452 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-oauth-serving-cert\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.883158 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.883134 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-console-config\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.883290 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.883267 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-service-ca\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.883361 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.883267 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-oauth-serving-cert\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.884707 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.884678 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/896f2237-4122-4df3-bd94-d33c09a3923a-console-oauth-config\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.884873 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.884844 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/896f2237-4122-4df3-bd94-d33c09a3923a-console-serving-cert\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.890896 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.890877 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mgb8\" (UniqueName: \"kubernetes.io/projected/896f2237-4122-4df3-bd94-d33c09a3923a-kube-api-access-4mgb8\") pod \"console-68ffbf4cc4-cscnw\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:37.935694 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:37.935645 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:38.005983 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:38.005878 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:38.143425 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:38.143392 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68ffbf4cc4-cscnw"] Jul 10 22:23:38.146529 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:23:38.146504 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896f2237_4122_4df3_bd94_d33c09a3923a.slice/crio-bd9a27a9cf0579b9c18cbb42766a8584cd8a8d45c4731bf7e2c27a4104b731c0 WatchSource:0}: Error finding container bd9a27a9cf0579b9c18cbb42766a8584cd8a8d45c4731bf7e2c27a4104b731c0: Status 404 returned error can't find the container with id bd9a27a9cf0579b9c18cbb42766a8584cd8a8d45c4731bf7e2c27a4104b731c0 Jul 10 22:23:38.943489 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:38.943427 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:39.044972 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:39.044932 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:39.081866 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:39.081825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68ffbf4cc4-cscnw" event={"ID":"896f2237-4122-4df3-bd94-d33c09a3923a","Type":"ContainerStarted","Data":"bd9a27a9cf0579b9c18cbb42766a8584cd8a8d45c4731bf7e2c27a4104b731c0"} Jul 10 22:23:40.984604 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:40.984576 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:41.088256 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:41.088225 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:41.090647 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:41.090618 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68ffbf4cc4-cscnw" event={"ID":"896f2237-4122-4df3-bd94-d33c09a3923a","Type":"ContainerStarted","Data":"126026a953fa7c647c1d1af210bbaf604032c24d52961d9984cf745683e73635"} Jul 10 22:23:41.110175 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:41.110121 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68ffbf4cc4-cscnw" podStartSLOduration=1.308745548 podStartE2EDuration="4.110100011s" totalImagesPullingTime="2.801354463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:41.109046551 +0000 UTC m=+84.941930933" watchObservedRunningTime="2026-07-10 22:23:41.110100011 +0000 UTC m=+84.942984395" Jul 10 22:23:48.006201 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:48.006147 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:48.006705 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:48.006231 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:48.010885 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:48.010862 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:48.121429 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:48.121402 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:23:49.279056 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:49.279017 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:49.333094 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:49.333051 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:51.105643 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:51.105605 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:51.205981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:51.205940 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:53.473521 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:53.473486 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:53.524192 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:53.524152 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:55.331649 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:55.331615 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:55.386548 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:55.386516 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:55.672420 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:55.672335 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:55.725199 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:55.725162 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:57.477087 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:57.477049 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:23:57.578389 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:23:57.578351 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:00.048955 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:00.048929 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vzlrd" Jul 10 22:24:00.061840 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:00.061814 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68ffbf4cc4-cscnw"] Jul 10 22:24:04.271308 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:04.271262 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:04.335339 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:04.335309 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:04.452383 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:04.452338 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:04.509459 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:04.509428 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:06.286884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:06.286837 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:06.347607 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:06.347580 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:13.416005 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:13.415965 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:13.466520 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:13.466486 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:24.424876 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:24.424843 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-84fbb_38a4df0c-b54e-4152-868e-f71e9f3e7e32/serve-healthcheck-canary/0.log" Jul 10 22:24:25.086580 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.086514 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68ffbf4cc4-cscnw" podUID="896f2237-4122-4df3-bd94-d33c09a3923a" containerName="console" containerID="cri-o://126026a953fa7c647c1d1af210bbaf604032c24d52961d9984cf745683e73635" gracePeriod=15 Jul 10 22:24:25.251256 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.251224 2565 generic.go:356] "Generic (PLEG): container finished" podID="896f2237-4122-4df3-bd94-d33c09a3923a" containerID="126026a953fa7c647c1d1af210bbaf604032c24d52961d9984cf745683e73635" exitCode=0 Jul 10 22:24:25.251428 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.251273 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68ffbf4cc4-cscnw" event={"ID":"896f2237-4122-4df3-bd94-d33c09a3923a","Type":"ContainerDied","Data":"126026a953fa7c647c1d1af210bbaf604032c24d52961d9984cf745683e73635"} Jul 10 22:24:25.322078 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.322051 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:24:25.422930 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.422873 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/896f2237-4122-4df3-bd94-d33c09a3923a-console-serving-cert\") pod \"896f2237-4122-4df3-bd94-d33c09a3923a\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " Jul 10 22:24:25.422930 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.422908 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/896f2237-4122-4df3-bd94-d33c09a3923a-console-oauth-config\") pod \"896f2237-4122-4df3-bd94-d33c09a3923a\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " Jul 10 22:24:25.423086 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.422933 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-service-ca\") pod \"896f2237-4122-4df3-bd94-d33c09a3923a\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " Jul 10 22:24:25.423086 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.422987 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-console-config\") pod \"896f2237-4122-4df3-bd94-d33c09a3923a\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " Jul 10 22:24:25.423086 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.423024 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-oauth-serving-cert\") pod \"896f2237-4122-4df3-bd94-d33c09a3923a\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " Jul 10 22:24:25.423086 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.423056 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mgb8\" (UniqueName: \"kubernetes.io/projected/896f2237-4122-4df3-bd94-d33c09a3923a-kube-api-access-4mgb8\") pod \"896f2237-4122-4df3-bd94-d33c09a3923a\" (UID: \"896f2237-4122-4df3-bd94-d33c09a3923a\") " Jul 10 22:24:25.423362 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.423332 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-console-config" (OuterVolumeSpecName: "console-config") pod "896f2237-4122-4df3-bd94-d33c09a3923a" (UID: "896f2237-4122-4df3-bd94-d33c09a3923a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:24:25.423423 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.423353 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-service-ca" (OuterVolumeSpecName: "service-ca") pod "896f2237-4122-4df3-bd94-d33c09a3923a" (UID: "896f2237-4122-4df3-bd94-d33c09a3923a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:24:25.423423 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.423372 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "896f2237-4122-4df3-bd94-d33c09a3923a" (UID: "896f2237-4122-4df3-bd94-d33c09a3923a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:24:25.425186 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.425161 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896f2237-4122-4df3-bd94-d33c09a3923a-kube-api-access-4mgb8" (OuterVolumeSpecName: "kube-api-access-4mgb8") pod "896f2237-4122-4df3-bd94-d33c09a3923a" (UID: "896f2237-4122-4df3-bd94-d33c09a3923a"). InnerVolumeSpecName "kube-api-access-4mgb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:24:25.425523 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.425403 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/896f2237-4122-4df3-bd94-d33c09a3923a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "896f2237-4122-4df3-bd94-d33c09a3923a" (UID: "896f2237-4122-4df3-bd94-d33c09a3923a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:24:25.425523 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.425417 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/896f2237-4122-4df3-bd94-d33c09a3923a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "896f2237-4122-4df3-bd94-d33c09a3923a" (UID: "896f2237-4122-4df3-bd94-d33c09a3923a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:24:25.524025 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.524003 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/896f2237-4122-4df3-bd94-d33c09a3923a-console-oauth-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:24:25.524025 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.524022 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-service-ca\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:24:25.524141 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.524032 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-console-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:24:25.524141 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.524040 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/896f2237-4122-4df3-bd94-d33c09a3923a-oauth-serving-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:24:25.524141 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.524049 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mgb8\" (UniqueName: \"kubernetes.io/projected/896f2237-4122-4df3-bd94-d33c09a3923a-kube-api-access-4mgb8\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:24:25.524141 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:25.524057 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/896f2237-4122-4df3-bd94-d33c09a3923a-console-serving-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:24:26.255846 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:26.255821 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68ffbf4cc4-cscnw" Jul 10 22:24:26.256018 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:26.255820 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68ffbf4cc4-cscnw" event={"ID":"896f2237-4122-4df3-bd94-d33c09a3923a","Type":"ContainerDied","Data":"bd9a27a9cf0579b9c18cbb42766a8584cd8a8d45c4731bf7e2c27a4104b731c0"} Jul 10 22:24:26.256018 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:26.255954 2565 scope.go:117] "RemoveContainer" containerID="126026a953fa7c647c1d1af210bbaf604032c24d52961d9984cf745683e73635" Jul 10 22:24:26.272367 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:26.272343 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68ffbf4cc4-cscnw"] Jul 10 22:24:26.276121 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:26.276098 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68ffbf4cc4-cscnw"] Jul 10 22:24:26.506085 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:26.505980 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:26.608412 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:26.608381 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:26.680721 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:26.680696 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:27.689702 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:27.689670 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="896f2237-4122-4df3-bd94-d33c09a3923a" path="/var/lib/kubelet/pods/896f2237-4122-4df3-bd94-d33c09a3923a/volumes" Jul 10 22:24:46.833359 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:46.833326 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:46.883187 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:46.883151 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:49.706752 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:49.706716 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:58.760082 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:58.760047 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:58.809239 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:58.809198 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:59.102329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.102251 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dd59c8566-xr5nr"] Jul 10 22:24:59.102487 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.102475 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="896f2237-4122-4df3-bd94-d33c09a3923a" containerName="console" Jul 10 22:24:59.102526 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.102488 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="896f2237-4122-4df3-bd94-d33c09a3923a" containerName="console" Jul 10 22:24:59.102558 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.102536 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="896f2237-4122-4df3-bd94-d33c09a3923a" containerName="console" Jul 10 22:24:59.106739 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.106721 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.110262 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.110236 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-f5j8x\"" Jul 10 22:24:59.110262 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.110255 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 22:24:59.111120 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.111097 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 22:24:59.111120 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.111102 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 22:24:59.111288 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.111179 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 22:24:59.111709 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.111692 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 22:24:59.111832 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.111780 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 22:24:59.112040 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.112020 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 22:24:59.118342 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.118324 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 22:24:59.125245 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.125221 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dd59c8566-xr5nr"] Jul 10 22:24:59.155070 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.155037 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/535acecf-ae50-4ad5-81f9-ead7d79df068-console-serving-cert\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.155222 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.155076 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-trusted-ca-bundle\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.155222 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.155105 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-console-config\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.155222 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.155124 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-service-ca\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.155222 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.155146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-oauth-serving-cert\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.155222 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.155176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x44bb\" (UniqueName: \"kubernetes.io/projected/535acecf-ae50-4ad5-81f9-ead7d79df068-kube-api-access-x44bb\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.155222 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.155193 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/535acecf-ae50-4ad5-81f9-ead7d79df068-console-oauth-config\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.195373 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.195341 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:59.256479 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.256445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/535acecf-ae50-4ad5-81f9-ead7d79df068-console-serving-cert\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.256479 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.256480 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-trusted-ca-bundle\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.256694 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.256508 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-console-config\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.256694 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.256534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-service-ca\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.256694 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.256558 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-oauth-serving-cert\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.256694 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.256602 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x44bb\" (UniqueName: \"kubernetes.io/projected/535acecf-ae50-4ad5-81f9-ead7d79df068-kube-api-access-x44bb\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.256694 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.256630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/535acecf-ae50-4ad5-81f9-ead7d79df068-console-oauth-config\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.257408 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.257378 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-console-config\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.257543 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.257378 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-service-ca\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.257543 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.257450 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-oauth-serving-cert\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.257543 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.257541 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-trusted-ca-bundle\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.259076 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.259054 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/535acecf-ae50-4ad5-81f9-ead7d79df068-console-oauth-config\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.259172 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.259122 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/535acecf-ae50-4ad5-81f9-ead7d79df068-console-serving-cert\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.264991 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.264972 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x44bb\" (UniqueName: \"kubernetes.io/projected/535acecf-ae50-4ad5-81f9-ead7d79df068-kube-api-access-x44bb\") pod \"console-5dd59c8566-xr5nr\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.300937 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.300900 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:59.397180 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.397105 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:59.416044 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.416020 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:24:59.543190 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.543167 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dd59c8566-xr5nr"] Jul 10 22:24:59.545867 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:24:59.545838 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535acecf_ae50_4ad5_81f9_ead7d79df068.slice/crio-6f2b7bee71fa3d3c4fad25d3c41829300194bf459526af2d6668937e484e9cf8 WatchSource:0}: Error finding container 6f2b7bee71fa3d3c4fad25d3c41829300194bf459526af2d6668937e484e9cf8: Status 404 returned error can't find the container with id 6f2b7bee71fa3d3c4fad25d3c41829300194bf459526af2d6668937e484e9cf8 Jul 10 22:24:59.895243 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.895210 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:24:59.949762 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:24:59.949722 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:00.265901 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:00.265871 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:00.369468 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:00.369434 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dd59c8566-xr5nr" event={"ID":"535acecf-ae50-4ad5-81f9-ead7d79df068","Type":"ContainerStarted","Data":"ac1d36fd2f05bdb74c79c72b2d351325fed05a6336f48dc7ce400d58f7988d44"} Jul 10 22:25:00.369625 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:00.369502 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dd59c8566-xr5nr" event={"ID":"535acecf-ae50-4ad5-81f9-ead7d79df068","Type":"ContainerStarted","Data":"6f2b7bee71fa3d3c4fad25d3c41829300194bf459526af2d6668937e484e9cf8"} Jul 10 22:25:00.390414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:00.389401 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5dd59c8566-xr5nr" podStartSLOduration=1.389375262 podStartE2EDuration="1.389375262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:25:00.386995106 +0000 UTC m=+164.219879491" watchObservedRunningTime="2026-07-10 22:25:00.389375262 +0000 UTC m=+164.222259645" Jul 10 22:25:04.277070 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:04.277038 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:04.377527 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:04.377490 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:09.417116 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:09.417027 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:25:09.417116 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:09.417114 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:25:09.422040 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:09.422016 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:25:10.403998 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:10.403968 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:25:10.551935 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:10.551900 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:10.606004 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:10.605959 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:10.665216 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:10.665130 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:23.208637 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:23.208603 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:23.302593 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:23.302555 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:23.804531 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:23.804327 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk"] Jul 10 22:25:23.808849 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:23.808831 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" Jul 10 22:25:23.811115 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:23.811095 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 22:25:23.811185 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:23.811117 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4dh4r\"" Jul 10 22:25:23.811933 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:23.811914 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 22:25:23.816420 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:23.816385 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk"] Jul 10 22:25:23.931338 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:23.931299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b568aec9-292d-4b5b-a59c-2cada67e9638-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk\" (UID: \"b568aec9-292d-4b5b-a59c-2cada67e9638\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" Jul 10 22:25:23.931517 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:23.931369 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b568aec9-292d-4b5b-a59c-2cada67e9638-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk\" (UID: \"b568aec9-292d-4b5b-a59c-2cada67e9638\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" Jul 10 22:25:23.931517 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:23.931418 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw6gv\" (UniqueName: \"kubernetes.io/projected/b568aec9-292d-4b5b-a59c-2cada67e9638-kube-api-access-zw6gv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk\" (UID: \"b568aec9-292d-4b5b-a59c-2cada67e9638\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" Jul 10 22:25:24.032496 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:24.032454 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b568aec9-292d-4b5b-a59c-2cada67e9638-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk\" (UID: \"b568aec9-292d-4b5b-a59c-2cada67e9638\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" Jul 10 22:25:24.032692 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:24.032510 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw6gv\" (UniqueName: \"kubernetes.io/projected/b568aec9-292d-4b5b-a59c-2cada67e9638-kube-api-access-zw6gv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk\" (UID: \"b568aec9-292d-4b5b-a59c-2cada67e9638\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" Jul 10 22:25:24.032692 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:24.032530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b568aec9-292d-4b5b-a59c-2cada67e9638-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk\" (UID: \"b568aec9-292d-4b5b-a59c-2cada67e9638\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" Jul 10 22:25:24.032980 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:24.032957 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b568aec9-292d-4b5b-a59c-2cada67e9638-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk\" (UID: \"b568aec9-292d-4b5b-a59c-2cada67e9638\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" Jul 10 22:25:24.033050 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:24.032958 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b568aec9-292d-4b5b-a59c-2cada67e9638-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk\" (UID: \"b568aec9-292d-4b5b-a59c-2cada67e9638\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" Jul 10 22:25:24.042356 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:24.042326 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw6gv\" (UniqueName: \"kubernetes.io/projected/b568aec9-292d-4b5b-a59c-2cada67e9638-kube-api-access-zw6gv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk\" (UID: \"b568aec9-292d-4b5b-a59c-2cada67e9638\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" Jul 10 22:25:24.119255 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:24.119191 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" Jul 10 22:25:24.240350 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:24.240321 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk"] Jul 10 22:25:24.242957 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:25:24.242929 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb568aec9_292d_4b5b_a59c_2cada67e9638.slice/crio-32061461f57d6f62f5258afa946541610de356ad0c38be341290e89d88e56a6e WatchSource:0}: Error finding container 32061461f57d6f62f5258afa946541610de356ad0c38be341290e89d88e56a6e: Status 404 returned error can't find the container with id 32061461f57d6f62f5258afa946541610de356ad0c38be341290e89d88e56a6e Jul 10 22:25:24.452291 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:24.452211 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" event={"ID":"b568aec9-292d-4b5b-a59c-2cada67e9638","Type":"ContainerStarted","Data":"32061461f57d6f62f5258afa946541610de356ad0c38be341290e89d88e56a6e"} Jul 10 22:25:30.474980 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:30.474946 2565 generic.go:356] "Generic (PLEG): container finished" podID="b568aec9-292d-4b5b-a59c-2cada67e9638" containerID="8f583cf230e32b7e1c531f7a48fa097d4b862d19c3916ca1dceea1ed63123d0a" exitCode=0 Jul 10 22:25:30.475466 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:30.475000 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" event={"ID":"b568aec9-292d-4b5b-a59c-2cada67e9638","Type":"ContainerDied","Data":"8f583cf230e32b7e1c531f7a48fa097d4b862d19c3916ca1dceea1ed63123d0a"} Jul 10 22:25:32.487739 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:32.487701 2565 generic.go:356] "Generic (PLEG): container finished" podID="b568aec9-292d-4b5b-a59c-2cada67e9638" containerID="9627cfdbd6650492fdab849629a57849b4d037470b4cc695636957be8bbff38f" exitCode=0 Jul 10 22:25:32.488126 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:32.487744 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" event={"ID":"b568aec9-292d-4b5b-a59c-2cada67e9638","Type":"ContainerDied","Data":"9627cfdbd6650492fdab849629a57849b4d037470b4cc695636957be8bbff38f"} Jul 10 22:25:39.515011 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:39.514974 2565 generic.go:356] "Generic (PLEG): container finished" podID="b568aec9-292d-4b5b-a59c-2cada67e9638" containerID="36f11827af2f57b239bf0db011ef8aa5e2f592534d1aeb6e75b4ad3ad81cb485" exitCode=0 Jul 10 22:25:39.515011 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:39.515016 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" event={"ID":"b568aec9-292d-4b5b-a59c-2cada67e9638","Type":"ContainerDied","Data":"36f11827af2f57b239bf0db011ef8aa5e2f592534d1aeb6e75b4ad3ad81cb485"} Jul 10 22:25:40.644520 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:40.644495 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" Jul 10 22:25:40.767669 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:40.767635 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b568aec9-292d-4b5b-a59c-2cada67e9638-util\") pod \"b568aec9-292d-4b5b-a59c-2cada67e9638\" (UID: \"b568aec9-292d-4b5b-a59c-2cada67e9638\") " Jul 10 22:25:40.767861 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:40.767697 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw6gv\" (UniqueName: \"kubernetes.io/projected/b568aec9-292d-4b5b-a59c-2cada67e9638-kube-api-access-zw6gv\") pod \"b568aec9-292d-4b5b-a59c-2cada67e9638\" (UID: \"b568aec9-292d-4b5b-a59c-2cada67e9638\") " Jul 10 22:25:40.767861 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:40.767720 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b568aec9-292d-4b5b-a59c-2cada67e9638-bundle\") pod \"b568aec9-292d-4b5b-a59c-2cada67e9638\" (UID: \"b568aec9-292d-4b5b-a59c-2cada67e9638\") " Jul 10 22:25:40.768558 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:40.768506 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b568aec9-292d-4b5b-a59c-2cada67e9638-bundle" (OuterVolumeSpecName: "bundle") pod "b568aec9-292d-4b5b-a59c-2cada67e9638" (UID: "b568aec9-292d-4b5b-a59c-2cada67e9638"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:25:40.769895 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:40.769862 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b568aec9-292d-4b5b-a59c-2cada67e9638-kube-api-access-zw6gv" (OuterVolumeSpecName: "kube-api-access-zw6gv") pod "b568aec9-292d-4b5b-a59c-2cada67e9638" (UID: "b568aec9-292d-4b5b-a59c-2cada67e9638"). InnerVolumeSpecName "kube-api-access-zw6gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:25:40.772638 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:40.772612 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b568aec9-292d-4b5b-a59c-2cada67e9638-util" (OuterVolumeSpecName: "util") pod "b568aec9-292d-4b5b-a59c-2cada67e9638" (UID: "b568aec9-292d-4b5b-a59c-2cada67e9638"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:25:40.868487 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:40.868449 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw6gv\" (UniqueName: \"kubernetes.io/projected/b568aec9-292d-4b5b-a59c-2cada67e9638-kube-api-access-zw6gv\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:25:40.868487 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:40.868481 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b568aec9-292d-4b5b-a59c-2cada67e9638-bundle\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:25:40.868487 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:40.868490 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b568aec9-292d-4b5b-a59c-2cada67e9638-util\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:25:41.522941 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:41.522910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" event={"ID":"b568aec9-292d-4b5b-a59c-2cada67e9638","Type":"ContainerDied","Data":"32061461f57d6f62f5258afa946541610de356ad0c38be341290e89d88e56a6e"} Jul 10 22:25:41.522941 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:41.522941 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754087bzk" Jul 10 22:25:41.523148 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:41.522955 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32061461f57d6f62f5258afa946541610de356ad0c38be341290e89d88e56a6e" Jul 10 22:25:44.783488 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:44.783454 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:45.604222 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:45.604182 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:46.534638 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:46.534602 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:48.789842 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:48.789806 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:48.889828 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:48.889787 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:50.143122 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:50.143087 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:50.307805 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:50.307771 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:50.871314 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:50.871278 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:51.016182 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:51.016150 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:51.957590 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:51.957555 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:52.540153 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:52.540117 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:54.560848 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:54.560810 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:54.662187 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:54.662151 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:55.603740 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:55.603705 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:55.704258 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:55.704227 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:58.753504 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:58.753460 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:25:58.810639 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:25:58.810605 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:01.327535 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:01.327498 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:02.777639 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.777604 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-z25qs"] Jul 10 22:26:02.778023 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.777848 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b568aec9-292d-4b5b-a59c-2cada67e9638" containerName="util" Jul 10 22:26:02.778023 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.777859 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b568aec9-292d-4b5b-a59c-2cada67e9638" containerName="util" Jul 10 22:26:02.778023 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.777867 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b568aec9-292d-4b5b-a59c-2cada67e9638" containerName="pull" Jul 10 22:26:02.778023 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.777872 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b568aec9-292d-4b5b-a59c-2cada67e9638" containerName="pull" Jul 10 22:26:02.778023 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.777877 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b568aec9-292d-4b5b-a59c-2cada67e9638" containerName="extract" Jul 10 22:26:02.778023 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.777882 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b568aec9-292d-4b5b-a59c-2cada67e9638" containerName="extract" Jul 10 22:26:02.778023 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.777926 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b568aec9-292d-4b5b-a59c-2cada67e9638" containerName="extract" Jul 10 22:26:02.781232 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.781216 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-z25qs" Jul 10 22:26:02.783596 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.783576 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9q4sd\"" Jul 10 22:26:02.783757 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.783742 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 22:26:02.784383 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.784369 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 22:26:02.788627 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.788606 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-z25qs"] Jul 10 22:26:02.917498 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.917450 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7jl\" (UniqueName: \"kubernetes.io/projected/7d3b1d43-2879-4aca-a4ad-f66b700e234c-kube-api-access-7w7jl\") pod \"cert-manager-55b4d954d5-z25qs\" (UID: \"7d3b1d43-2879-4aca-a4ad-f66b700e234c\") " pod="cert-manager/cert-manager-55b4d954d5-z25qs" Jul 10 22:26:02.917498 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:02.917500 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d3b1d43-2879-4aca-a4ad-f66b700e234c-bound-sa-token\") pod \"cert-manager-55b4d954d5-z25qs\" (UID: \"7d3b1d43-2879-4aca-a4ad-f66b700e234c\") " pod="cert-manager/cert-manager-55b4d954d5-z25qs" Jul 10 22:26:03.018851 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:03.018811 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7jl\" (UniqueName: \"kubernetes.io/projected/7d3b1d43-2879-4aca-a4ad-f66b700e234c-kube-api-access-7w7jl\") pod \"cert-manager-55b4d954d5-z25qs\" (UID: \"7d3b1d43-2879-4aca-a4ad-f66b700e234c\") " pod="cert-manager/cert-manager-55b4d954d5-z25qs" Jul 10 22:26:03.018851 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:03.018851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d3b1d43-2879-4aca-a4ad-f66b700e234c-bound-sa-token\") pod \"cert-manager-55b4d954d5-z25qs\" (UID: \"7d3b1d43-2879-4aca-a4ad-f66b700e234c\") " pod="cert-manager/cert-manager-55b4d954d5-z25qs" Jul 10 22:26:03.028039 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:03.027973 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d3b1d43-2879-4aca-a4ad-f66b700e234c-bound-sa-token\") pod \"cert-manager-55b4d954d5-z25qs\" (UID: \"7d3b1d43-2879-4aca-a4ad-f66b700e234c\") " pod="cert-manager/cert-manager-55b4d954d5-z25qs" Jul 10 22:26:03.028126 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:03.028085 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7jl\" (UniqueName: \"kubernetes.io/projected/7d3b1d43-2879-4aca-a4ad-f66b700e234c-kube-api-access-7w7jl\") pod \"cert-manager-55b4d954d5-z25qs\" (UID: \"7d3b1d43-2879-4aca-a4ad-f66b700e234c\") " pod="cert-manager/cert-manager-55b4d954d5-z25qs" Jul 10 22:26:03.091363 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:03.091322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-z25qs" Jul 10 22:26:03.217781 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:03.217750 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-z25qs"] Jul 10 22:26:03.221538 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:26:03.221510 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d3b1d43_2879_4aca_a4ad_f66b700e234c.slice/crio-3867dd5f1c3271cfb6a3572d832c6e911ac4e2d5c1a9f0c606d9425f490ec41b WatchSource:0}: Error finding container 3867dd5f1c3271cfb6a3572d832c6e911ac4e2d5c1a9f0c606d9425f490ec41b: Status 404 returned error can't find the container with id 3867dd5f1c3271cfb6a3572d832c6e911ac4e2d5c1a9f0c606d9425f490ec41b Jul 10 22:26:03.598401 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:03.598367 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-z25qs" event={"ID":"7d3b1d43-2879-4aca-a4ad-f66b700e234c","Type":"ContainerStarted","Data":"3867dd5f1c3271cfb6a3572d832c6e911ac4e2d5c1a9f0c606d9425f490ec41b"} Jul 10 22:26:06.611678 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:06.611639 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-z25qs" event={"ID":"7d3b1d43-2879-4aca-a4ad-f66b700e234c","Type":"ContainerStarted","Data":"2196b2238833b6a546c86567173ff1aec63e63a92803fb8782ad55b7c81b4238"} Jul 10 22:26:06.626724 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:06.626684 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-z25qs" podStartSLOduration=1.696721564 podStartE2EDuration="4.626671278s" totalImagesPullingTime="2.929949714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:26:06.626086249 +0000 UTC m=+230.458970636" watchObservedRunningTime="2026-07-10 22:26:06.626671278 +0000 UTC m=+230.459555661" Jul 10 22:26:13.194374 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:13.194339 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:13.296212 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:13.296179 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:15.420554 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.420519 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9"] Jul 10 22:26:15.422675 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.422659 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9" Jul 10 22:26:15.425012 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.424989 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 22:26:15.425139 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.424991 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 22:26:15.425694 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.425679 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-nzljt\"" Jul 10 22:26:15.434522 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.434501 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9"] Jul 10 22:26:15.505902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.505867 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkrhp\" (UniqueName: \"kubernetes.io/projected/d6a6b966-a9e5-49f7-bb52-44f0e861bc07-kube-api-access-kkrhp\") pod \"openshift-lws-operator-fd8ccff4c-4f8x9\" (UID: \"d6a6b966-a9e5-49f7-bb52-44f0e861bc07\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9" Jul 10 22:26:15.505902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.505908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6a6b966-a9e5-49f7-bb52-44f0e861bc07-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4f8x9\" (UID: \"d6a6b966-a9e5-49f7-bb52-44f0e861bc07\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9" Jul 10 22:26:15.606767 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.606707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkrhp\" (UniqueName: \"kubernetes.io/projected/d6a6b966-a9e5-49f7-bb52-44f0e861bc07-kube-api-access-kkrhp\") pod \"openshift-lws-operator-fd8ccff4c-4f8x9\" (UID: \"d6a6b966-a9e5-49f7-bb52-44f0e861bc07\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9" Jul 10 22:26:15.606767 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.606777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6a6b966-a9e5-49f7-bb52-44f0e861bc07-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4f8x9\" (UID: \"d6a6b966-a9e5-49f7-bb52-44f0e861bc07\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9" Jul 10 22:26:15.607180 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.607156 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6a6b966-a9e5-49f7-bb52-44f0e861bc07-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4f8x9\" (UID: \"d6a6b966-a9e5-49f7-bb52-44f0e861bc07\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9" Jul 10 22:26:15.614803 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.614772 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkrhp\" (UniqueName: \"kubernetes.io/projected/d6a6b966-a9e5-49f7-bb52-44f0e861bc07-kube-api-access-kkrhp\") pod \"openshift-lws-operator-fd8ccff4c-4f8x9\" (UID: \"d6a6b966-a9e5-49f7-bb52-44f0e861bc07\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9" Jul 10 22:26:15.731460 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.731422 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9" Jul 10 22:26:15.858305 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:15.858271 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9"] Jul 10 22:26:15.861693 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:26:15.861651 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6a6b966_a9e5_49f7_bb52_44f0e861bc07.slice/crio-682b94fe524c7b319218041380048d6af3d423f9bd59b431a101eabcd217be92 WatchSource:0}: Error finding container 682b94fe524c7b319218041380048d6af3d423f9bd59b431a101eabcd217be92: Status 404 returned error can't find the container with id 682b94fe524c7b319218041380048d6af3d423f9bd59b431a101eabcd217be92 Jul 10 22:26:16.647184 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:16.647131 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9" event={"ID":"d6a6b966-a9e5-49f7-bb52-44f0e861bc07","Type":"ContainerStarted","Data":"682b94fe524c7b319218041380048d6af3d423f9bd59b431a101eabcd217be92"} Jul 10 22:26:18.655193 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:18.655103 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9" event={"ID":"d6a6b966-a9e5-49f7-bb52-44f0e861bc07","Type":"ContainerStarted","Data":"27dacec0e94816a6a72cf499e2b27eccb9b63a147f7fa424e469bc19ac5614af"} Jul 10 22:26:18.671491 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:18.671427 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4f8x9" podStartSLOduration=1.284171801 podStartE2EDuration="3.671407613s" totalImagesPullingTime="2.387235812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:26:18.670849819 +0000 UTC m=+242.503734202" watchObservedRunningTime="2026-07-10 22:26:18.671407613 +0000 UTC m=+242.504291996" Jul 10 22:26:19.297289 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:19.297249 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:19.398411 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:19.398372 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:21.257951 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.257921 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:21.337273 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.337245 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj"] Jul 10 22:26:21.339498 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.339481 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" Jul 10 22:26:21.341751 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.341730 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4dh4r\"" Jul 10 22:26:21.341891 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.341760 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 22:26:21.342624 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.342609 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 22:26:21.344390 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.344373 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c91f601-63d2-47dd-bb30-5fadd9a2c459-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj\" (UID: \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" Jul 10 22:26:21.344460 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.344412 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c91f601-63d2-47dd-bb30-5fadd9a2c459-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj\" (UID: \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" Jul 10 22:26:21.344460 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.344438 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4jf\" (UniqueName: \"kubernetes.io/projected/9c91f601-63d2-47dd-bb30-5fadd9a2c459-kube-api-access-5d4jf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj\" (UID: \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" Jul 10 22:26:21.351055 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.351035 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj"] Jul 10 22:26:21.445635 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.445596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c91f601-63d2-47dd-bb30-5fadd9a2c459-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj\" (UID: \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" Jul 10 22:26:21.445635 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.445636 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4jf\" (UniqueName: \"kubernetes.io/projected/9c91f601-63d2-47dd-bb30-5fadd9a2c459-kube-api-access-5d4jf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj\" (UID: \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" Jul 10 22:26:21.445878 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.445698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c91f601-63d2-47dd-bb30-5fadd9a2c459-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj\" (UID: \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" Jul 10 22:26:21.446063 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.446038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c91f601-63d2-47dd-bb30-5fadd9a2c459-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj\" (UID: \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" Jul 10 22:26:21.446104 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.446052 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c91f601-63d2-47dd-bb30-5fadd9a2c459-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj\" (UID: \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" Jul 10 22:26:21.453359 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.453337 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4jf\" (UniqueName: \"kubernetes.io/projected/9c91f601-63d2-47dd-bb30-5fadd9a2c459-kube-api-access-5d4jf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj\" (UID: \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" Jul 10 22:26:21.647989 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.647884 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" Jul 10 22:26:21.779748 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:21.779718 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj"] Jul 10 22:26:21.782384 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:26:21.782351 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c91f601_63d2_47dd_bb30_5fadd9a2c459.slice/crio-c48a22204631fada516e119597ef561bfb7356b75d796b19b4af7fb3bad17a9a WatchSource:0}: Error finding container c48a22204631fada516e119597ef561bfb7356b75d796b19b4af7fb3bad17a9a: Status 404 returned error can't find the container with id c48a22204631fada516e119597ef561bfb7356b75d796b19b4af7fb3bad17a9a Jul 10 22:26:22.671488 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:22.671454 2565 generic.go:356] "Generic (PLEG): container finished" podID="9c91f601-63d2-47dd-bb30-5fadd9a2c459" containerID="6e26df73d8fd94de8c7e34ac2914b8f87cdcd565a20ec7414664293fc69ba0c1" exitCode=0 Jul 10 22:26:22.671869 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:22.671521 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" event={"ID":"9c91f601-63d2-47dd-bb30-5fadd9a2c459","Type":"ContainerDied","Data":"6e26df73d8fd94de8c7e34ac2914b8f87cdcd565a20ec7414664293fc69ba0c1"} Jul 10 22:26:22.671869 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:22.671565 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" event={"ID":"9c91f601-63d2-47dd-bb30-5fadd9a2c459","Type":"ContainerStarted","Data":"c48a22204631fada516e119597ef561bfb7356b75d796b19b4af7fb3bad17a9a"} Jul 10 22:26:23.677026 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:23.676989 2565 generic.go:356] "Generic (PLEG): container finished" podID="9c91f601-63d2-47dd-bb30-5fadd9a2c459" containerID="d53ff28f611ab7d044872cb616f07e94446f4f433f407335c1246e2c94e2bb0f" exitCode=0 Jul 10 22:26:23.677401 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:23.677073 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" event={"ID":"9c91f601-63d2-47dd-bb30-5fadd9a2c459","Type":"ContainerDied","Data":"d53ff28f611ab7d044872cb616f07e94446f4f433f407335c1246e2c94e2bb0f"} Jul 10 22:26:24.682349 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:24.682318 2565 generic.go:356] "Generic (PLEG): container finished" podID="9c91f601-63d2-47dd-bb30-5fadd9a2c459" containerID="de7fd5b60b2dcdcd8e7205da8d3db7a1df3293801835e62b3f67446a18d2fe0c" exitCode=0 Jul 10 22:26:24.682741 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:24.682372 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" event={"ID":"9c91f601-63d2-47dd-bb30-5fadd9a2c459","Type":"ContainerDied","Data":"de7fd5b60b2dcdcd8e7205da8d3db7a1df3293801835e62b3f67446a18d2fe0c"} Jul 10 22:26:25.810111 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:25.810089 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" Jul 10 22:26:25.873985 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:25.873957 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c91f601-63d2-47dd-bb30-5fadd9a2c459-util\") pod \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\" (UID: \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\") " Jul 10 22:26:25.874138 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:25.874058 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d4jf\" (UniqueName: \"kubernetes.io/projected/9c91f601-63d2-47dd-bb30-5fadd9a2c459-kube-api-access-5d4jf\") pod \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\" (UID: \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\") " Jul 10 22:26:25.874138 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:25.874090 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c91f601-63d2-47dd-bb30-5fadd9a2c459-bundle\") pod \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\" (UID: \"9c91f601-63d2-47dd-bb30-5fadd9a2c459\") " Jul 10 22:26:25.874769 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:25.874743 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c91f601-63d2-47dd-bb30-5fadd9a2c459-bundle" (OuterVolumeSpecName: "bundle") pod "9c91f601-63d2-47dd-bb30-5fadd9a2c459" (UID: "9c91f601-63d2-47dd-bb30-5fadd9a2c459"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:26:25.876058 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:25.876031 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c91f601-63d2-47dd-bb30-5fadd9a2c459-kube-api-access-5d4jf" (OuterVolumeSpecName: "kube-api-access-5d4jf") pod "9c91f601-63d2-47dd-bb30-5fadd9a2c459" (UID: "9c91f601-63d2-47dd-bb30-5fadd9a2c459"). InnerVolumeSpecName "kube-api-access-5d4jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:26:25.880032 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:25.880006 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c91f601-63d2-47dd-bb30-5fadd9a2c459-util" (OuterVolumeSpecName: "util") pod "9c91f601-63d2-47dd-bb30-5fadd9a2c459" (UID: "9c91f601-63d2-47dd-bb30-5fadd9a2c459"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:26:25.974508 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:25.974477 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d4jf\" (UniqueName: \"kubernetes.io/projected/9c91f601-63d2-47dd-bb30-5fadd9a2c459-kube-api-access-5d4jf\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:26:25.974508 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:25.974508 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c91f601-63d2-47dd-bb30-5fadd9a2c459-bundle\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:26:25.974679 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:25.974519 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c91f601-63d2-47dd-bb30-5fadd9a2c459-util\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:26:26.690807 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:26.690751 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" event={"ID":"9c91f601-63d2-47dd-bb30-5fadd9a2c459","Type":"ContainerDied","Data":"c48a22204631fada516e119597ef561bfb7356b75d796b19b4af7fb3bad17a9a"} Jul 10 22:26:26.690807 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:26.690770 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftwffj" Jul 10 22:26:26.691013 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:26.690823 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c48a22204631fada516e119597ef561bfb7356b75d796b19b4af7fb3bad17a9a" Jul 10 22:26:33.799526 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:33.799490 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:33.899683 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:33.899649 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:34.428820 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.428764 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:34.647682 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.647646 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:34.749649 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.749615 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:34.770938 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.770908 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8"] Jul 10 22:26:34.771141 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.771128 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c91f601-63d2-47dd-bb30-5fadd9a2c459" containerName="pull" Jul 10 22:26:34.771183 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.771142 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c91f601-63d2-47dd-bb30-5fadd9a2c459" containerName="pull" Jul 10 22:26:34.771183 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.771161 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c91f601-63d2-47dd-bb30-5fadd9a2c459" containerName="util" Jul 10 22:26:34.771183 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.771168 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c91f601-63d2-47dd-bb30-5fadd9a2c459" containerName="util" Jul 10 22:26:34.771183 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.771174 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c91f601-63d2-47dd-bb30-5fadd9a2c459" containerName="extract" Jul 10 22:26:34.771183 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.771180 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c91f601-63d2-47dd-bb30-5fadd9a2c459" containerName="extract" Jul 10 22:26:34.771330 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.771226 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c91f601-63d2-47dd-bb30-5fadd9a2c459" containerName="extract" Jul 10 22:26:34.773582 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.773564 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.776403 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.776378 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 22:26:34.776527 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.776417 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 22:26:34.776527 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.776383 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hzgd7\"" Jul 10 22:26:34.776633 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.776542 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 22:26:34.776633 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.776544 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 22:26:34.788543 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.788522 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8"] Jul 10 22:26:34.828409 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.828376 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-manifests\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.828768 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.828421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.828768 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.828488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.828768 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.828518 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-tmp\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.828768 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.828549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p82bb\" (UniqueName: \"kubernetes.io/projected/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-kube-api-access-p82bb\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.899917 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.899881 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:34.929512 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.929478 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.929623 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.929526 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.929623 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.929547 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-tmp\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.929833 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.929720 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82bb\" (UniqueName: \"kubernetes.io/projected/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-kube-api-access-p82bb\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.929833 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.929770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-manifests\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.930112 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.930085 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-manifests\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.931894 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.931872 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-tmp\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.932038 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.932021 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.932158 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.932140 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:34.938447 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:34.938420 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p82bb\" (UniqueName: \"kubernetes.io/projected/ecd095a4-0f72-40d2-98c1-1e6f7421a3ea-kube-api-access-p82bb\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t88g8\" (UID: \"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:35.009730 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:35.009642 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:35.083641 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:35.083605 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:35.100824 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:35.100784 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:35.201177 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:35.201150 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:35.221339 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:35.221319 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8"] Jul 10 22:26:35.223971 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:26:35.223934 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd095a4_0f72_40d2_98c1_1e6f7421a3ea.slice/crio-8dae0251cd820710e408e30ef8031c53dce3c95da4e221286911e82c1ae72f0d WatchSource:0}: Error finding container 8dae0251cd820710e408e30ef8031c53dce3c95da4e221286911e82c1ae72f0d: Status 404 returned error can't find the container with id 8dae0251cd820710e408e30ef8031c53dce3c95da4e221286911e82c1ae72f0d Jul 10 22:26:35.301511 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:35.301429 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:35.721960 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:35.721925 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" event={"ID":"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea","Type":"ContainerStarted","Data":"8dae0251cd820710e408e30ef8031c53dce3c95da4e221286911e82c1ae72f0d"} Jul 10 22:26:38.736531 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:38.736484 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" event={"ID":"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea","Type":"ContainerStarted","Data":"e56a3839d5bf30dfbf94da47dc7f464ff8273c12d6af10add8c9df49341d8915"} Jul 10 22:26:39.741632 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:39.741593 2565 generic.go:356] "Generic (PLEG): container finished" podID="ecd095a4-0f72-40d2-98c1-1e6f7421a3ea" containerID="e56a3839d5bf30dfbf94da47dc7f464ff8273c12d6af10add8c9df49341d8915" exitCode=0 Jul 10 22:26:39.742034 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:39.741661 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" event={"ID":"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea","Type":"ContainerDied","Data":"e56a3839d5bf30dfbf94da47dc7f464ff8273c12d6af10add8c9df49341d8915"} Jul 10 22:26:40.752542 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:40.752505 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" event={"ID":"ecd095a4-0f72-40d2-98c1-1e6f7421a3ea","Type":"ContainerStarted","Data":"8e70610b08fd45416c8b07662a232e237bf49c7ab3ab6d7f4af8b642f9a8655c"} Jul 10 22:26:40.753016 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:40.752659 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:40.775018 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:40.774967 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" podStartSLOduration=2.438911647 podStartE2EDuration="6.774951487s" totalImagesPullingTime="3.317396185s" totalInitContainerRuntime="1.018643655s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:26:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:26:40.773913211 +0000 UTC m=+264.606797595" watchObservedRunningTime="2026-07-10 22:26:40.774951487 +0000 UTC m=+264.607835870" Jul 10 22:26:41.506788 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:41.506730 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:41.600172 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:41.600139 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:45.979403 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:45.979374 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:46.085385 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:46.085348 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:51.759193 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:51.759155 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t88g8" Jul 10 22:26:53.627532 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:53.627501 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:53.777981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:53.777923 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:54.315787 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.315746 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:54.318210 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.318183 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6"] Jul 10 22:26:54.325461 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.325440 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" Jul 10 22:26:54.329403 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.329382 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 22:26:54.329879 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.329861 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 22:26:54.330121 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.330107 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 22:26:54.332189 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.332161 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 22:26:54.337584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.337557 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6"] Jul 10 22:26:54.374499 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.374464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a335eb-0ac8-49e3-8ea1-31491736893b-tls-certs\") pod \"kube-auth-proxy-fc969b7d8-ps4x6\" (UID: \"27a335eb-0ac8-49e3-8ea1-31491736893b\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" Jul 10 22:26:54.374499 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.374497 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27a335eb-0ac8-49e3-8ea1-31491736893b-tmp\") pod \"kube-auth-proxy-fc969b7d8-ps4x6\" (UID: \"27a335eb-0ac8-49e3-8ea1-31491736893b\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" Jul 10 22:26:54.374691 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.374521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqtdz\" (UniqueName: \"kubernetes.io/projected/27a335eb-0ac8-49e3-8ea1-31491736893b-kube-api-access-zqtdz\") pod \"kube-auth-proxy-fc969b7d8-ps4x6\" (UID: \"27a335eb-0ac8-49e3-8ea1-31491736893b\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" Jul 10 22:26:54.475319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.475280 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27a335eb-0ac8-49e3-8ea1-31491736893b-tmp\") pod \"kube-auth-proxy-fc969b7d8-ps4x6\" (UID: \"27a335eb-0ac8-49e3-8ea1-31491736893b\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" Jul 10 22:26:54.475508 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.475334 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqtdz\" (UniqueName: \"kubernetes.io/projected/27a335eb-0ac8-49e3-8ea1-31491736893b-kube-api-access-zqtdz\") pod \"kube-auth-proxy-fc969b7d8-ps4x6\" (UID: \"27a335eb-0ac8-49e3-8ea1-31491736893b\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" Jul 10 22:26:54.475508 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.475446 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a335eb-0ac8-49e3-8ea1-31491736893b-tls-certs\") pod \"kube-auth-proxy-fc969b7d8-ps4x6\" (UID: \"27a335eb-0ac8-49e3-8ea1-31491736893b\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" Jul 10 22:26:54.477601 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.477576 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27a335eb-0ac8-49e3-8ea1-31491736893b-tmp\") pod \"kube-auth-proxy-fc969b7d8-ps4x6\" (UID: \"27a335eb-0ac8-49e3-8ea1-31491736893b\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" Jul 10 22:26:54.477901 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.477884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a335eb-0ac8-49e3-8ea1-31491736893b-tls-certs\") pod \"kube-auth-proxy-fc969b7d8-ps4x6\" (UID: \"27a335eb-0ac8-49e3-8ea1-31491736893b\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" Jul 10 22:26:54.506091 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.506058 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqtdz\" (UniqueName: \"kubernetes.io/projected/27a335eb-0ac8-49e3-8ea1-31491736893b-kube-api-access-zqtdz\") pod \"kube-auth-proxy-fc969b7d8-ps4x6\" (UID: \"27a335eb-0ac8-49e3-8ea1-31491736893b\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" Jul 10 22:26:54.530358 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.530330 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pgqgb"] Jul 10 22:26:54.532615 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.532600 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" Jul 10 22:26:54.536595 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.536577 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 22:26:54.536843 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.536830 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-22m8k\"" Jul 10 22:26:54.564973 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.564952 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pgqgb"] Jul 10 22:26:54.566180 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.566124 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:54.576294 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.576240 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdx7x\" (UniqueName: \"kubernetes.io/projected/8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b-kube-api-access-gdx7x\") pod \"odh-model-controller-7998c67dc4-pgqgb\" (UID: \"8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" Jul 10 22:26:54.576294 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.576292 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b-cert\") pod \"odh-model-controller-7998c67dc4-pgqgb\" (UID: \"8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" Jul 10 22:26:54.634850 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.634817 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" Jul 10 22:26:54.656522 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.656490 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:54.676714 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.676682 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdx7x\" (UniqueName: \"kubernetes.io/projected/8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b-kube-api-access-gdx7x\") pod \"odh-model-controller-7998c67dc4-pgqgb\" (UID: \"8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" Jul 10 22:26:54.676863 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.676726 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b-cert\") pod \"odh-model-controller-7998c67dc4-pgqgb\" (UID: \"8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" Jul 10 22:26:54.676863 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:26:54.676853 2565 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 22:26:54.676946 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:26:54.676918 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b-cert podName:8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b nodeName:}" failed. No retries permitted until 2026-07-10 22:26:55.176899848 +0000 UTC m=+279.009784208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b-cert") pod "odh-model-controller-7998c67dc4-pgqgb" (UID: "8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b") : secret "odh-model-controller-webhook-cert" not found Jul 10 22:26:54.688689 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.688659 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdx7x\" (UniqueName: \"kubernetes.io/projected/8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b-kube-api-access-gdx7x\") pod \"odh-model-controller-7998c67dc4-pgqgb\" (UID: \"8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" Jul 10 22:26:54.738597 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.738561 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:54.763739 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.763710 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6"] Jul 10 22:26:54.766282 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:26:54.766249 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a335eb_0ac8_49e3_8ea1_31491736893b.slice/crio-b72b72b25090b22fdcae12a528c75a5a752c9f7297a7fa865a1d563ffbbc8b8a WatchSource:0}: Error finding container b72b72b25090b22fdcae12a528c75a5a752c9f7297a7fa865a1d563ffbbc8b8a: Status 404 returned error can't find the container with id b72b72b25090b22fdcae12a528c75a5a752c9f7297a7fa865a1d563ffbbc8b8a Jul 10 22:26:54.803983 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.803950 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" event={"ID":"27a335eb-0ac8-49e3-8ea1-31491736893b","Type":"ContainerStarted","Data":"b72b72b25090b22fdcae12a528c75a5a752c9f7297a7fa865a1d563ffbbc8b8a"} Jul 10 22:26:54.914811 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:54.914713 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:55.179423 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.179337 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b-cert\") pod \"odh-model-controller-7998c67dc4-pgqgb\" (UID: \"8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" Jul 10 22:26:55.181708 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.181687 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b-cert\") pod \"odh-model-controller-7998c67dc4-pgqgb\" (UID: \"8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" Jul 10 22:26:55.352578 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.352545 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:55.442512 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.442426 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" Jul 10 22:26:55.493379 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.493346 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zvqhz"] Jul 10 22:26:55.496682 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.496657 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zvqhz" Jul 10 22:26:55.501311 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.500322 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 22:26:55.522907 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.522882 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zvqhz"] Jul 10 22:26:55.582377 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.582340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm6s8\" (UniqueName: \"kubernetes.io/projected/20e39149-1c1b-44d5-939d-88ab9382fec7-kube-api-access-dm6s8\") pod \"model-serving-api-54d89bd5fc-zvqhz\" (UID: \"20e39149-1c1b-44d5-939d-88ab9382fec7\") " pod="opendatahub/model-serving-api-54d89bd5fc-zvqhz" Jul 10 22:26:55.582546 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.582421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20e39149-1c1b-44d5-939d-88ab9382fec7-tls-certs\") pod \"model-serving-api-54d89bd5fc-zvqhz\" (UID: \"20e39149-1c1b-44d5-939d-88ab9382fec7\") " pod="opendatahub/model-serving-api-54d89bd5fc-zvqhz" Jul 10 22:26:55.590686 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.590616 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:55.621362 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.621330 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pgqgb"] Jul 10 22:26:55.627504 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:26:55.627473 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0bb4d2_6c19_49bd_b0cf_1f49619cfb0b.slice/crio-28d2f56936db1a482545b4dfacf8c86a2d9860a8d45e661077657a5fa61b110f WatchSource:0}: Error finding container 28d2f56936db1a482545b4dfacf8c86a2d9860a8d45e661077657a5fa61b110f: Status 404 returned error can't find the container with id 28d2f56936db1a482545b4dfacf8c86a2d9860a8d45e661077657a5fa61b110f Jul 10 22:26:55.634470 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.634442 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb"] Jul 10 22:26:55.637685 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.637659 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.640044 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.640019 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 22:26:55.640130 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.640048 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 22:26:55.640130 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.640099 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 22:26:55.640130 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.640109 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 22:26:55.640230 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.640184 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xh7nm\"" Jul 10 22:26:55.649329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.649305 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb"] Jul 10 22:26:55.674642 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.674573 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:55.683922 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.683885 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fl2j\" (UniqueName: \"kubernetes.io/projected/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-kube-api-access-6fl2j\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.684096 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.683946 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20e39149-1c1b-44d5-939d-88ab9382fec7-tls-certs\") pod \"model-serving-api-54d89bd5fc-zvqhz\" (UID: \"20e39149-1c1b-44d5-939d-88ab9382fec7\") " pod="opendatahub/model-serving-api-54d89bd5fc-zvqhz" Jul 10 22:26:55.684096 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.683973 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-local-certs\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.684096 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.684003 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-token\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.684096 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.684033 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.684274 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.684112 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-cacerts\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.684274 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.684157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm6s8\" (UniqueName: \"kubernetes.io/projected/20e39149-1c1b-44d5-939d-88ab9382fec7-kube-api-access-dm6s8\") pod \"model-serving-api-54d89bd5fc-zvqhz\" (UID: \"20e39149-1c1b-44d5-939d-88ab9382fec7\") " pod="opendatahub/model-serving-api-54d89bd5fc-zvqhz" Jul 10 22:26:55.684274 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.684203 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.684274 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.684231 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.687194 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.687168 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20e39149-1c1b-44d5-939d-88ab9382fec7-tls-certs\") pod \"model-serving-api-54d89bd5fc-zvqhz\" (UID: \"20e39149-1c1b-44d5-939d-88ab9382fec7\") " pod="opendatahub/model-serving-api-54d89bd5fc-zvqhz" Jul 10 22:26:55.697417 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.697349 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm6s8\" (UniqueName: \"kubernetes.io/projected/20e39149-1c1b-44d5-939d-88ab9382fec7-kube-api-access-dm6s8\") pod \"model-serving-api-54d89bd5fc-zvqhz\" (UID: \"20e39149-1c1b-44d5-939d-88ab9382fec7\") " pod="opendatahub/model-serving-api-54d89bd5fc-zvqhz" Jul 10 22:26:55.737813 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.737758 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:55.785421 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.785385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-local-certs\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.785613 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.785429 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-token\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.785613 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.785462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.785613 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.785494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-cacerts\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.785613 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.785542 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.785613 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.785568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.785885 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.785777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fl2j\" (UniqueName: \"kubernetes.io/projected/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-kube-api-access-6fl2j\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.786239 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.786211 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.788171 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.788147 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-local-certs\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.788414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.788327 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.788414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.788397 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-cacerts\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.788529 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.788507 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.793716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.793689 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fl2j\" (UniqueName: \"kubernetes.io/projected/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-kube-api-access-6fl2j\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.793837 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.793817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-token\") pod \"istiod-openshift-gateway-7574459656-5ghpb\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.809485 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.809456 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" event={"ID":"8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b","Type":"ContainerStarted","Data":"28d2f56936db1a482545b4dfacf8c86a2d9860a8d45e661077657a5fa61b110f"} Jul 10 22:26:55.813694 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.813662 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zvqhz" Jul 10 22:26:55.950554 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.950193 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:26:55.968050 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:55.968027 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zvqhz"] Jul 10 22:26:55.970996 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:26:55.970963 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e39149_1c1b_44d5_939d_88ab9382fec7.slice/crio-667f8ce8157313359d0397ac38386479f32b59522c0b9408cc7f88b61f7e0fb4 WatchSource:0}: Error finding container 667f8ce8157313359d0397ac38386479f32b59522c0b9408cc7f88b61f7e0fb4: Status 404 returned error can't find the container with id 667f8ce8157313359d0397ac38386479f32b59522c0b9408cc7f88b61f7e0fb4 Jul 10 22:26:56.101543 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:56.101515 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb"] Jul 10 22:26:56.103149 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:26:56.103112 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fd18c8_da72_4a1a_9cc7_6d07e1b2758b.slice/crio-31d3a27616f4e6f55c1dc576319620792f77991a6cfe4885bf51adf8a68f81b5 WatchSource:0}: Error finding container 31d3a27616f4e6f55c1dc576319620792f77991a6cfe4885bf51adf8a68f81b5: Status 404 returned error can't find the container with id 31d3a27616f4e6f55c1dc576319620792f77991a6cfe4885bf51adf8a68f81b5 Jul 10 22:26:56.730760 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:56.730715 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:26:56.817527 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:56.817439 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" event={"ID":"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b","Type":"ContainerStarted","Data":"31d3a27616f4e6f55c1dc576319620792f77991a6cfe4885bf51adf8a68f81b5"} Jul 10 22:26:56.820586 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:26:56.820545 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zvqhz" event={"ID":"20e39149-1c1b-44d5-939d-88ab9382fec7","Type":"ContainerStarted","Data":"667f8ce8157313359d0397ac38386479f32b59522c0b9408cc7f88b61f7e0fb4"} Jul 10 22:27:00.567779 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:00.567726 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:27:00.568175 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:00.567842 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:27:00.842234 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:00.842132 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" event={"ID":"8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b","Type":"ContainerStarted","Data":"d2583b698db6d3a6e22e77e9e4168a54eb72014c5345e5d0b9130f30db0e6562"} Jul 10 22:27:00.842395 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:00.842236 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" Jul 10 22:27:00.843754 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:00.843723 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" event={"ID":"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b","Type":"ContainerStarted","Data":"f071d2684b77b6326cd28d9dcfed1685e05801bfbc608b955d664e6d715dedbe"} Jul 10 22:27:00.843917 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:00.843772 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:27:00.845116 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:00.845085 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zvqhz" event={"ID":"20e39149-1c1b-44d5-939d-88ab9382fec7","Type":"ContainerStarted","Data":"f63ff6b4f5795daf14747dc0290fa6d24ac6beaf34c196960f11a0f0dc5ffcfe"} Jul 10 22:27:00.845234 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:00.845149 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-zvqhz" Jul 10 22:27:00.846549 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:00.846528 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" event={"ID":"27a335eb-0ac8-49e3-8ea1-31491736893b","Type":"ContainerStarted","Data":"efa99a86fcba70514f7449cc4f8fc7c491a0bb731029c6d6a63e26d6ea75b087"} Jul 10 22:27:00.859701 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:00.859658 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" podStartSLOduration=1.8732398730000002 podStartE2EDuration="6.859643431s" totalImagesPullingTime="4.986403558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:27:00.858300494 +0000 UTC m=+284.691184878" watchObservedRunningTime="2026-07-10 22:27:00.859643431 +0000 UTC m=+284.692527814" Jul 10 22:27:00.873827 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:00.873755 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-fc969b7d8-ps4x6" podStartSLOduration=1.080458839 podStartE2EDuration="6.873739823s" totalImagesPullingTime="5.793280984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:27:00.87339962 +0000 UTC m=+284.706284004" watchObservedRunningTime="2026-07-10 22:27:00.873739823 +0000 UTC m=+284.706624207" Jul 10 22:27:00.892868 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:00.892816 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" podStartSLOduration=1.4305771250000001 podStartE2EDuration="5.892776108s" totalImagesPullingTime="4.462198983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:27:00.891083543 +0000 UTC m=+284.723967927" watchObservedRunningTime="2026-07-10 22:27:00.892776108 +0000 UTC m=+284.725660491" Jul 10 22:27:00.912327 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:00.912272 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-zvqhz" podStartSLOduration=1.3229618539999999 podStartE2EDuration="5.912251653s" totalImagesPullingTime="4.589289799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:27:00.911129789 +0000 UTC m=+284.744014173" watchObservedRunningTime="2026-07-10 22:27:00.912251653 +0000 UTC m=+284.745136036" Jul 10 22:27:01.465320 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:01.465265 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:01.559942 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:01.559877 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:01.852379 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:01.852350 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-5ghpb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 22:27:01.852738 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:01.852399 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" podUID="d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 22:27:02.119141 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.119069 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj"] Jul 10 22:27:02.121773 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.121751 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.124362 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.124343 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dnvxv\"" Jul 10 22:27:02.142374 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.142350 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj"] Jul 10 22:27:02.151250 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.151217 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:02.245722 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.245689 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/897507dc-df36-48fc-a691-a9e28fec3394-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.245932 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.245735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2tjv\" (UniqueName: \"kubernetes.io/projected/897507dc-df36-48fc-a691-a9e28fec3394-kube-api-access-p2tjv\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.245932 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.245774 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.245932 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.245831 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.245932 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.245857 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.245932 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.245883 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.246103 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.245965 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/897507dc-df36-48fc-a691-a9e28fec3394-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.246103 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.245996 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.246103 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.246049 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/897507dc-df36-48fc-a691-a9e28fec3394-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.316353 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.316321 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:02.347114 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/897507dc-df36-48fc-a691-a9e28fec3394-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.347114 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.347329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347152 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/897507dc-df36-48fc-a691-a9e28fec3394-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.347329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347188 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/897507dc-df36-48fc-a691-a9e28fec3394-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.347329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2tjv\" (UniqueName: \"kubernetes.io/projected/897507dc-df36-48fc-a691-a9e28fec3394-kube-api-access-p2tjv\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.347329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347237 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.347329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.347329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347287 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.347329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347317 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.347974 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347693 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.347974 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347757 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.347974 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347856 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.347974 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347870 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/897507dc-df36-48fc-a691-a9e28fec3394-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.347974 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.347946 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.349632 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.349613 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/897507dc-df36-48fc-a691-a9e28fec3394-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.350021 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.350004 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/897507dc-df36-48fc-a691-a9e28fec3394-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.360291 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.360263 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2tjv\" (UniqueName: \"kubernetes.io/projected/897507dc-df36-48fc-a691-a9e28fec3394-kube-api-access-p2tjv\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.360483 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.360459 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/897507dc-df36-48fc-a691-a9e28fec3394-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678spvxj\" (UID: \"897507dc-df36-48fc-a691-a9e28fec3394\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.423184 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.423099 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:02.431673 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.431646 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:02.516356 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.516324 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:02.578761 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.578730 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj"] Jul 10 22:27:02.581051 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:27:02.581022 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897507dc_df36_48fc_a691_a9e28fec3394.slice/crio-4f79e1434cb9b025c10fcdea07b5bf37c501d841d79badd44b5b596f09573585 WatchSource:0}: Error finding container 4f79e1434cb9b025c10fcdea07b5bf37c501d841d79badd44b5b596f09573585: Status 404 returned error can't find the container with id 4f79e1434cb9b025c10fcdea07b5bf37c501d841d79badd44b5b596f09573585 Jul 10 22:27:02.855727 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:02.855684 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" event={"ID":"897507dc-df36-48fc-a691-a9e28fec3394","Type":"ContainerStarted","Data":"4f79e1434cb9b025c10fcdea07b5bf37c501d841d79badd44b5b596f09573585"} Jul 10 22:27:04.585093 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.585059 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:04.636860 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.636824 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm"] Jul 10 22:27:04.641320 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.641303 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" Jul 10 22:27:04.643755 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.643734 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-tc88x\"" Jul 10 22:27:04.643891 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.643734 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 22:27:04.652627 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.652606 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm"] Jul 10 22:27:04.694178 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.694147 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:04.772225 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.772194 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30f4ad53-fa0f-4a7b-af87-20b48ae7783e-cert\") pod \"kserve-controller-manager-d6f4b65b8-bj7fm\" (UID: \"30f4ad53-fa0f-4a7b-af87-20b48ae7783e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" Jul 10 22:27:04.772394 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.772236 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb9h8\" (UniqueName: \"kubernetes.io/projected/30f4ad53-fa0f-4a7b-af87-20b48ae7783e-kube-api-access-vb9h8\") pod \"kserve-controller-manager-d6f4b65b8-bj7fm\" (UID: \"30f4ad53-fa0f-4a7b-af87-20b48ae7783e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" Jul 10 22:27:04.852188 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.852105 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:27:04.873180 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.873147 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30f4ad53-fa0f-4a7b-af87-20b48ae7783e-cert\") pod \"kserve-controller-manager-d6f4b65b8-bj7fm\" (UID: \"30f4ad53-fa0f-4a7b-af87-20b48ae7783e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" Jul 10 22:27:04.873340 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.873188 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb9h8\" (UniqueName: \"kubernetes.io/projected/30f4ad53-fa0f-4a7b-af87-20b48ae7783e-kube-api-access-vb9h8\") pod \"kserve-controller-manager-d6f4b65b8-bj7fm\" (UID: \"30f4ad53-fa0f-4a7b-af87-20b48ae7783e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" Jul 10 22:27:04.873340 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:27:04.873295 2565 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 10 22:27:04.873421 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:27:04.873356 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f4ad53-fa0f-4a7b-af87-20b48ae7783e-cert podName:30f4ad53-fa0f-4a7b-af87-20b48ae7783e nodeName:}" failed. No retries permitted until 2026-07-10 22:27:05.373339142 +0000 UTC m=+289.206223503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/30f4ad53-fa0f-4a7b-af87-20b48ae7783e-cert") pod "kserve-controller-manager-d6f4b65b8-bj7fm" (UID: "30f4ad53-fa0f-4a7b-af87-20b48ae7783e") : secret "kserve-webhook-server-cert" not found Jul 10 22:27:04.898753 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.898722 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb9h8\" (UniqueName: \"kubernetes.io/projected/30f4ad53-fa0f-4a7b-af87-20b48ae7783e-kube-api-access-vb9h8\") pod \"kserve-controller-manager-d6f4b65b8-bj7fm\" (UID: \"30f4ad53-fa0f-4a7b-af87-20b48ae7783e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" Jul 10 22:27:04.964085 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:04.964050 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:05.285592 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:05.285555 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:05.377920 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:05.377883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30f4ad53-fa0f-4a7b-af87-20b48ae7783e-cert\") pod \"kserve-controller-manager-d6f4b65b8-bj7fm\" (UID: \"30f4ad53-fa0f-4a7b-af87-20b48ae7783e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" Jul 10 22:27:05.380397 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:05.380359 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30f4ad53-fa0f-4a7b-af87-20b48ae7783e-cert\") pod \"kserve-controller-manager-d6f4b65b8-bj7fm\" (UID: \"30f4ad53-fa0f-4a7b-af87-20b48ae7783e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" Jul 10 22:27:05.551780 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:05.551695 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" Jul 10 22:27:05.690650 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:27:05.690615 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f4ad53_fa0f_4a7b_af87_20b48ae7783e.slice/crio-5da100b51e7e6a03ba0c7e7fa7daf6ddb08b90c6336a2021ec655d951ce9f676 WatchSource:0}: Error finding container 5da100b51e7e6a03ba0c7e7fa7daf6ddb08b90c6336a2021ec655d951ce9f676: Status 404 returned error can't find the container with id 5da100b51e7e6a03ba0c7e7fa7daf6ddb08b90c6336a2021ec655d951ce9f676 Jul 10 22:27:05.692311 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:05.692283 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm"] Jul 10 22:27:05.869637 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:05.869538 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" event={"ID":"30f4ad53-fa0f-4a7b-af87-20b48ae7783e","Type":"ContainerStarted","Data":"5da100b51e7e6a03ba0c7e7fa7daf6ddb08b90c6336a2021ec655d951ce9f676"} Jul 10 22:27:05.885192 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:05.885166 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:05.972996 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:05.972964 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:06.928356 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:06.928322 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb"] Jul 10 22:27:06.928838 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:06.928530 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" podUID="d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" containerName="discovery" containerID="cri-o://f071d2684b77b6326cd28d9dcfed1685e05801bfbc608b955d664e6d715dedbe" gracePeriod=30 Jul 10 22:27:07.172119 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.172096 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:27:07.295557 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.295525 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-csr-ca-configmap\") pod \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " Jul 10 22:27:07.295725 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.295565 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-cacerts\") pod \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " Jul 10 22:27:07.295725 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.295610 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-local-certs\") pod \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " Jul 10 22:27:07.295725 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.295637 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fl2j\" (UniqueName: \"kubernetes.io/projected/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-kube-api-access-6fl2j\") pod \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " Jul 10 22:27:07.295725 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.295657 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-token\") pod \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " Jul 10 22:27:07.295725 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.295681 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-kubeconfig\") pod \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " Jul 10 22:27:07.295997 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.295727 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-csr-dns-cert\") pod \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\" (UID: \"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b\") " Jul 10 22:27:07.296055 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.295999 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" (UID: "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:27:07.298042 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.298009 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-kube-api-access-6fl2j" (OuterVolumeSpecName: "kube-api-access-6fl2j") pod "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" (UID: "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b"). InnerVolumeSpecName "kube-api-access-6fl2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:27:07.298172 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.298052 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" (UID: "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:27:07.298327 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.298300 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-cacerts" (OuterVolumeSpecName: "cacerts") pod "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" (UID: "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:27:07.298327 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.298326 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-local-certs" (OuterVolumeSpecName: "local-certs") pod "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" (UID: "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:27:07.298463 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.298368 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" (UID: "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:27:07.298463 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.298377 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" (UID: "d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:27:07.396958 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.396920 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-csr-dns-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:27:07.396958 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.396952 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-csr-ca-configmap\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:27:07.396958 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.396963 2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-cacerts\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:27:07.397185 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.396972 2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-local-certs\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:27:07.397185 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.396983 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fl2j\" (UniqueName: \"kubernetes.io/projected/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-kube-api-access-6fl2j\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:27:07.397185 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.396990 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-token\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:27:07.397185 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.396998 2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b-istio-kubeconfig\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:27:07.879244 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.879207 2565 generic.go:356] "Generic (PLEG): container finished" podID="d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" containerID="f071d2684b77b6326cd28d9dcfed1685e05801bfbc608b955d664e6d715dedbe" exitCode=0 Jul 10 22:27:07.879427 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.879264 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" Jul 10 22:27:07.879427 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.879282 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" event={"ID":"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b","Type":"ContainerDied","Data":"f071d2684b77b6326cd28d9dcfed1685e05801bfbc608b955d664e6d715dedbe"} Jul 10 22:27:07.879427 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.879338 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb" event={"ID":"d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b","Type":"ContainerDied","Data":"31d3a27616f4e6f55c1dc576319620792f77991a6cfe4885bf51adf8a68f81b5"} Jul 10 22:27:07.879427 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.879370 2565 scope.go:117] "RemoveContainer" containerID="f071d2684b77b6326cd28d9dcfed1685e05801bfbc608b955d664e6d715dedbe" Jul 10 22:27:07.898189 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.898155 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb"] Jul 10 22:27:07.902329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:07.902308 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5ghpb"] Jul 10 22:27:08.150215 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:08.150007 2565 scope.go:117] "RemoveContainer" containerID="f071d2684b77b6326cd28d9dcfed1685e05801bfbc608b955d664e6d715dedbe" Jul 10 22:27:08.150483 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:27:08.150457 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f071d2684b77b6326cd28d9dcfed1685e05801bfbc608b955d664e6d715dedbe\": container with ID starting with f071d2684b77b6326cd28d9dcfed1685e05801bfbc608b955d664e6d715dedbe not found: ID does not exist" containerID="f071d2684b77b6326cd28d9dcfed1685e05801bfbc608b955d664e6d715dedbe" Jul 10 22:27:08.150529 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:08.150495 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f071d2684b77b6326cd28d9dcfed1685e05801bfbc608b955d664e6d715dedbe"} err="failed to get container status \"f071d2684b77b6326cd28d9dcfed1685e05801bfbc608b955d664e6d715dedbe\": rpc error: code = NotFound desc = could not find container \"f071d2684b77b6326cd28d9dcfed1685e05801bfbc608b955d664e6d715dedbe\": container with ID starting with f071d2684b77b6326cd28d9dcfed1685e05801bfbc608b955d664e6d715dedbe not found: ID does not exist" Jul 10 22:27:08.885591 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:08.885554 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" event={"ID":"30f4ad53-fa0f-4a7b-af87-20b48ae7783e","Type":"ContainerStarted","Data":"a71b4999f83bd1bdde4f10d72cdca5e6664561fccfaf2b748b622148ac07028f"} Jul 10 22:27:08.885816 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:08.885661 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" Jul 10 22:27:08.901640 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:08.901586 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" podStartSLOduration=2.278988205 podStartE2EDuration="4.901571379s" totalImagesPullingTime="2.622583174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:27:08.900211983 +0000 UTC m=+292.733096380" watchObservedRunningTime="2026-07-10 22:27:08.901571379 +0000 UTC m=+292.734455752" Jul 10 22:27:09.690761 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:09.690728 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" path="/var/lib/kubelet/pods/d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b/volumes" Jul 10 22:27:11.852808 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:11.852763 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-pgqgb" Jul 10 22:27:11.854737 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:11.854713 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-zvqhz" Jul 10 22:27:18.182253 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:18.182225 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:27:18.182679 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:18.182231 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:27:18.189106 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:18.189082 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 22:27:18.222304 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:18.222265 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:27:18.222414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:18.222336 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:27:18.222414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:18.222367 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:27:18.929539 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:18.929505 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" event={"ID":"897507dc-df36-48fc-a691-a9e28fec3394","Type":"ContainerStarted","Data":"92801488bc18b37b4f7e04cffd0ba3fb07293fd449b402642ccba9c8fa637755"} Jul 10 22:27:18.951295 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:18.951244 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" podStartSLOduration=1.312391777 podStartE2EDuration="16.951229401s" totalImagesPullingTime="15.638837624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:27:18.95041932 +0000 UTC m=+302.783303704" watchObservedRunningTime="2026-07-10 22:27:18.951229401 +0000 UTC m=+302.784113784" Jul 10 22:27:19.096138 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:19.096108 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:19.432058 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:19.432026 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:19.436667 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:19.436639 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:19.933886 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:19.933859 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:19.934836 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:19.934812 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678spvxj" Jul 10 22:27:20.143688 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:20.143655 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:21.041156 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:21.041121 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:21.103608 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:21.103575 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:39.896771 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:39.896741 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bj7fm" Jul 10 22:27:55.087146 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.087113 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:55.191600 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.191548 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:55.195522 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.195495 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmqxg"] Jul 10 22:27:55.195776 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.195765 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" containerName="discovery" Jul 10 22:27:55.195832 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.195778 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" containerName="discovery" Jul 10 22:27:55.195883 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.195867 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3fd18c8-da72-4a1a-9cc7-6d07e1b2758b" containerName="discovery" Jul 10 22:27:55.204556 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.204532 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kmqxg" Jul 10 22:27:55.208070 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.208049 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 22:27:55.208317 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.208296 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-dkfgm\"" Jul 10 22:27:55.208820 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.208553 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 22:27:55.209380 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.209357 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmqxg"] Jul 10 22:27:55.256849 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.256818 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:55.279441 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.279412 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzr8r\" (UniqueName: \"kubernetes.io/projected/a095de90-dd5c-4f89-b7dc-4167c9b72ee9-kube-api-access-mzr8r\") pod \"kuadrant-operator-catalog-kmqxg\" (UID: \"a095de90-dd5c-4f89-b7dc-4167c9b72ee9\") " pod="kuadrant-system/kuadrant-operator-catalog-kmqxg" Jul 10 22:27:55.380020 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.379751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzr8r\" (UniqueName: \"kubernetes.io/projected/a095de90-dd5c-4f89-b7dc-4167c9b72ee9-kube-api-access-mzr8r\") pod \"kuadrant-operator-catalog-kmqxg\" (UID: \"a095de90-dd5c-4f89-b7dc-4167c9b72ee9\") " pod="kuadrant-system/kuadrant-operator-catalog-kmqxg" Jul 10 22:27:55.388268 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.388241 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzr8r\" (UniqueName: \"kubernetes.io/projected/a095de90-dd5c-4f89-b7dc-4167c9b72ee9-kube-api-access-mzr8r\") pod \"kuadrant-operator-catalog-kmqxg\" (UID: \"a095de90-dd5c-4f89-b7dc-4167c9b72ee9\") " pod="kuadrant-system/kuadrant-operator-catalog-kmqxg" Jul 10 22:27:55.515813 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.515741 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kmqxg" Jul 10 22:27:55.559230 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.559197 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmqxg"] Jul 10 22:27:55.646684 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.646654 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmqxg"] Jul 10 22:27:55.649509 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:27:55.649478 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda095de90_dd5c_4f89_b7dc_4167c9b72ee9.slice/crio-b2c3ba5441f42a3b03e7fa1ee923e1a2a1dd845df655828c91b081fd4fd099dd WatchSource:0}: Error finding container b2c3ba5441f42a3b03e7fa1ee923e1a2a1dd845df655828c91b081fd4fd099dd: Status 404 returned error can't find the container with id b2c3ba5441f42a3b03e7fa1ee923e1a2a1dd845df655828c91b081fd4fd099dd Jul 10 22:27:55.650955 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.650935 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 22:27:55.769534 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.769501 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zrvs8"] Jul 10 22:27:55.774406 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.774390 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zrvs8" Jul 10 22:27:55.780568 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.780542 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zrvs8"] Jul 10 22:27:55.885288 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.885254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2gg8\" (UniqueName: \"kubernetes.io/projected/99836a7c-8438-49b1-80f6-a34a30e2aa2b-kube-api-access-z2gg8\") pod \"kuadrant-operator-catalog-zrvs8\" (UID: \"99836a7c-8438-49b1-80f6-a34a30e2aa2b\") " pod="kuadrant-system/kuadrant-operator-catalog-zrvs8" Jul 10 22:27:55.986653 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.986617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2gg8\" (UniqueName: \"kubernetes.io/projected/99836a7c-8438-49b1-80f6-a34a30e2aa2b-kube-api-access-z2gg8\") pod \"kuadrant-operator-catalog-zrvs8\" (UID: \"99836a7c-8438-49b1-80f6-a34a30e2aa2b\") " pod="kuadrant-system/kuadrant-operator-catalog-zrvs8" Jul 10 22:27:55.993029 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.992998 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:55.995474 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:55.995447 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2gg8\" (UniqueName: \"kubernetes.io/projected/99836a7c-8438-49b1-80f6-a34a30e2aa2b-kube-api-access-z2gg8\") pod \"kuadrant-operator-catalog-zrvs8\" (UID: \"99836a7c-8438-49b1-80f6-a34a30e2aa2b\") " pod="kuadrant-system/kuadrant-operator-catalog-zrvs8" Jul 10 22:27:56.080750 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:56.080715 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kmqxg" event={"ID":"a095de90-dd5c-4f89-b7dc-4167c9b72ee9","Type":"ContainerStarted","Data":"b2c3ba5441f42a3b03e7fa1ee923e1a2a1dd845df655828c91b081fd4fd099dd"} Jul 10 22:27:56.084866 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:56.084849 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zrvs8" Jul 10 22:27:56.207257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:56.207177 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:27:56.223167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:56.223137 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zrvs8"] Jul 10 22:27:56.246050 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:27:56.245983 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99836a7c_8438_49b1_80f6_a34a30e2aa2b.slice/crio-d05a8348c339f700ef4b1933ca4d851a280c15ac70fa6edd463350a23949567c WatchSource:0}: Error finding container d05a8348c339f700ef4b1933ca4d851a280c15ac70fa6edd463350a23949567c: Status 404 returned error can't find the container with id d05a8348c339f700ef4b1933ca4d851a280c15ac70fa6edd463350a23949567c Jul 10 22:27:57.088162 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:57.088123 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zrvs8" event={"ID":"99836a7c-8438-49b1-80f6-a34a30e2aa2b","Type":"ContainerStarted","Data":"d05a8348c339f700ef4b1933ca4d851a280c15ac70fa6edd463350a23949567c"} Jul 10 22:27:58.095144 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:58.095107 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kmqxg" event={"ID":"a095de90-dd5c-4f89-b7dc-4167c9b72ee9","Type":"ContainerStarted","Data":"1a33a67aadaa52371dfdbe3c8dba7652dfdaf9692245c984d84061a841c138c1"} Jul 10 22:27:58.095537 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:58.095169 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-kmqxg" podUID="a095de90-dd5c-4f89-b7dc-4167c9b72ee9" containerName="registry-server" containerID="cri-o://1a33a67aadaa52371dfdbe3c8dba7652dfdaf9692245c984d84061a841c138c1" gracePeriod=2 Jul 10 22:27:58.096705 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:58.096679 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zrvs8" event={"ID":"99836a7c-8438-49b1-80f6-a34a30e2aa2b","Type":"ContainerStarted","Data":"eb8ceceab5ac6c74bba655f83d3ae2cfe8ab70a0e77671b137aa4fee387e8c94"} Jul 10 22:27:58.110214 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:58.110167 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-kmqxg" podStartSLOduration=1.056615701 podStartE2EDuration="3.110152202s" totalImagesPullingTime="2.053536501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:27:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:27:58.109725885 +0000 UTC m=+341.942610269" watchObservedRunningTime="2026-07-10 22:27:58.110152202 +0000 UTC m=+341.943036585" Jul 10 22:27:58.124069 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:58.124026 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zrvs8" podStartSLOduration=1.470990293 podStartE2EDuration="3.124011294s" totalImagesPullingTime="1.653021001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:27:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:27:58.122810638 +0000 UTC m=+341.955695023" watchObservedRunningTime="2026-07-10 22:27:58.124011294 +0000 UTC m=+341.956895701" Jul 10 22:27:58.337291 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:58.337265 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kmqxg" Jul 10 22:27:58.404426 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:58.404391 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzr8r\" (UniqueName: \"kubernetes.io/projected/a095de90-dd5c-4f89-b7dc-4167c9b72ee9-kube-api-access-mzr8r\") pod \"a095de90-dd5c-4f89-b7dc-4167c9b72ee9\" (UID: \"a095de90-dd5c-4f89-b7dc-4167c9b72ee9\") " Jul 10 22:27:58.406533 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:58.406511 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a095de90-dd5c-4f89-b7dc-4167c9b72ee9-kube-api-access-mzr8r" (OuterVolumeSpecName: "kube-api-access-mzr8r") pod "a095de90-dd5c-4f89-b7dc-4167c9b72ee9" (UID: "a095de90-dd5c-4f89-b7dc-4167c9b72ee9"). InnerVolumeSpecName "kube-api-access-mzr8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:27:58.505472 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:58.505438 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzr8r\" (UniqueName: \"kubernetes.io/projected/a095de90-dd5c-4f89-b7dc-4167c9b72ee9-kube-api-access-mzr8r\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:27:59.102542 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:59.102504 2565 generic.go:356] "Generic (PLEG): container finished" podID="a095de90-dd5c-4f89-b7dc-4167c9b72ee9" containerID="1a33a67aadaa52371dfdbe3c8dba7652dfdaf9692245c984d84061a841c138c1" exitCode=0 Jul 10 22:27:59.103045 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:59.102548 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kmqxg" Jul 10 22:27:59.103045 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:59.102588 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kmqxg" event={"ID":"a095de90-dd5c-4f89-b7dc-4167c9b72ee9","Type":"ContainerDied","Data":"1a33a67aadaa52371dfdbe3c8dba7652dfdaf9692245c984d84061a841c138c1"} Jul 10 22:27:59.103045 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:59.102654 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kmqxg" event={"ID":"a095de90-dd5c-4f89-b7dc-4167c9b72ee9","Type":"ContainerDied","Data":"b2c3ba5441f42a3b03e7fa1ee923e1a2a1dd845df655828c91b081fd4fd099dd"} Jul 10 22:27:59.103045 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:59.102685 2565 scope.go:117] "RemoveContainer" containerID="1a33a67aadaa52371dfdbe3c8dba7652dfdaf9692245c984d84061a841c138c1" Jul 10 22:27:59.111815 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:59.111783 2565 scope.go:117] "RemoveContainer" containerID="1a33a67aadaa52371dfdbe3c8dba7652dfdaf9692245c984d84061a841c138c1" Jul 10 22:27:59.112104 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:27:59.112084 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a33a67aadaa52371dfdbe3c8dba7652dfdaf9692245c984d84061a841c138c1\": container with ID starting with 1a33a67aadaa52371dfdbe3c8dba7652dfdaf9692245c984d84061a841c138c1 not found: ID does not exist" containerID="1a33a67aadaa52371dfdbe3c8dba7652dfdaf9692245c984d84061a841c138c1" Jul 10 22:27:59.112160 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:59.112112 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a33a67aadaa52371dfdbe3c8dba7652dfdaf9692245c984d84061a841c138c1"} err="failed to get container status \"1a33a67aadaa52371dfdbe3c8dba7652dfdaf9692245c984d84061a841c138c1\": rpc error: code = NotFound desc = could not find container \"1a33a67aadaa52371dfdbe3c8dba7652dfdaf9692245c984d84061a841c138c1\": container with ID starting with 1a33a67aadaa52371dfdbe3c8dba7652dfdaf9692245c984d84061a841c138c1 not found: ID does not exist" Jul 10 22:27:59.122882 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:59.122863 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmqxg"] Jul 10 22:27:59.128562 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:59.128543 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmqxg"] Jul 10 22:27:59.690316 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:27:59.690283 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a095de90-dd5c-4f89-b7dc-4167c9b72ee9" path="/var/lib/kubelet/pods/a095de90-dd5c-4f89-b7dc-4167c9b72ee9/volumes" Jul 10 22:28:06.085370 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:06.085329 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-zrvs8" Jul 10 22:28:06.085370 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:06.085383 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-zrvs8" Jul 10 22:28:06.107695 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:06.107667 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-zrvs8" Jul 10 22:28:06.153380 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:06.153355 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-zrvs8" Jul 10 22:28:07.378077 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.378041 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w"] Jul 10 22:28:07.378534 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.378343 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a095de90-dd5c-4f89-b7dc-4167c9b72ee9" containerName="registry-server" Jul 10 22:28:07.378534 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.378354 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a095de90-dd5c-4f89-b7dc-4167c9b72ee9" containerName="registry-server" Jul 10 22:28:07.378534 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.378438 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a095de90-dd5c-4f89-b7dc-4167c9b72ee9" containerName="registry-server" Jul 10 22:28:07.381514 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.381497 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" Jul 10 22:28:07.383813 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.383770 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tp8vl\"" Jul 10 22:28:07.389420 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.389400 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w"] Jul 10 22:28:07.479242 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.479192 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22aca153-5823-443b-bd4e-c0009121fed9-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w\" (UID: \"22aca153-5823-443b-bd4e-c0009121fed9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" Jul 10 22:28:07.479433 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.479267 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22aca153-5823-443b-bd4e-c0009121fed9-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w\" (UID: \"22aca153-5823-443b-bd4e-c0009121fed9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" Jul 10 22:28:07.479433 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.479295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4pfp\" (UniqueName: \"kubernetes.io/projected/22aca153-5823-443b-bd4e-c0009121fed9-kube-api-access-r4pfp\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w\" (UID: \"22aca153-5823-443b-bd4e-c0009121fed9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" Jul 10 22:28:07.580671 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.580625 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22aca153-5823-443b-bd4e-c0009121fed9-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w\" (UID: \"22aca153-5823-443b-bd4e-c0009121fed9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" Jul 10 22:28:07.580671 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.580690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22aca153-5823-443b-bd4e-c0009121fed9-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w\" (UID: \"22aca153-5823-443b-bd4e-c0009121fed9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" Jul 10 22:28:07.580929 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.580786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4pfp\" (UniqueName: \"kubernetes.io/projected/22aca153-5823-443b-bd4e-c0009121fed9-kube-api-access-r4pfp\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w\" (UID: \"22aca153-5823-443b-bd4e-c0009121fed9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" Jul 10 22:28:07.581131 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.581110 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22aca153-5823-443b-bd4e-c0009121fed9-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w\" (UID: \"22aca153-5823-443b-bd4e-c0009121fed9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" Jul 10 22:28:07.581167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.581141 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22aca153-5823-443b-bd4e-c0009121fed9-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w\" (UID: \"22aca153-5823-443b-bd4e-c0009121fed9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" Jul 10 22:28:07.590230 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.590200 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4pfp\" (UniqueName: \"kubernetes.io/projected/22aca153-5823-443b-bd4e-c0009121fed9-kube-api-access-r4pfp\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w\" (UID: \"22aca153-5823-443b-bd4e-c0009121fed9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" Jul 10 22:28:07.690665 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.690583 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" Jul 10 22:28:07.825101 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.825045 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w"] Jul 10 22:28:07.828048 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:28:07.828014 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22aca153_5823_443b_bd4e_c0009121fed9.slice/crio-82daad859577159fe42af2c77be577f7a0542143e7ed4dc8e6be3e21654eaa3e WatchSource:0}: Error finding container 82daad859577159fe42af2c77be577f7a0542143e7ed4dc8e6be3e21654eaa3e: Status 404 returned error can't find the container with id 82daad859577159fe42af2c77be577f7a0542143e7ed4dc8e6be3e21654eaa3e Jul 10 22:28:07.973252 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.973217 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl"] Jul 10 22:28:07.976544 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.976521 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" Jul 10 22:28:07.991997 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:07.991930 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl"] Jul 10 22:28:08.084159 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.084122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmzdg\" (UniqueName: \"kubernetes.io/projected/e1accdaf-e3bd-4260-a715-737e35a214c1-kube-api-access-vmzdg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl\" (UID: \"e1accdaf-e3bd-4260-a715-737e35a214c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" Jul 10 22:28:08.084331 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.084183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1accdaf-e3bd-4260-a715-737e35a214c1-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl\" (UID: \"e1accdaf-e3bd-4260-a715-737e35a214c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" Jul 10 22:28:08.084331 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.084222 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1accdaf-e3bd-4260-a715-737e35a214c1-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl\" (UID: \"e1accdaf-e3bd-4260-a715-737e35a214c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" Jul 10 22:28:08.142884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.142852 2565 generic.go:356] "Generic (PLEG): container finished" podID="22aca153-5823-443b-bd4e-c0009121fed9" containerID="f0e270499c1c6f409e099b70a22c2a3e602bd36d56ca3a97ba6ec81f9dd4cb1d" exitCode=0 Jul 10 22:28:08.143060 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.142891 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" event={"ID":"22aca153-5823-443b-bd4e-c0009121fed9","Type":"ContainerDied","Data":"f0e270499c1c6f409e099b70a22c2a3e602bd36d56ca3a97ba6ec81f9dd4cb1d"} Jul 10 22:28:08.143060 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.142929 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" event={"ID":"22aca153-5823-443b-bd4e-c0009121fed9","Type":"ContainerStarted","Data":"82daad859577159fe42af2c77be577f7a0542143e7ed4dc8e6be3e21654eaa3e"} Jul 10 22:28:08.184983 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.184940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmzdg\" (UniqueName: \"kubernetes.io/projected/e1accdaf-e3bd-4260-a715-737e35a214c1-kube-api-access-vmzdg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl\" (UID: \"e1accdaf-e3bd-4260-a715-737e35a214c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" Jul 10 22:28:08.185170 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.184996 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1accdaf-e3bd-4260-a715-737e35a214c1-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl\" (UID: \"e1accdaf-e3bd-4260-a715-737e35a214c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" Jul 10 22:28:08.185170 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.185020 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1accdaf-e3bd-4260-a715-737e35a214c1-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl\" (UID: \"e1accdaf-e3bd-4260-a715-737e35a214c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" Jul 10 22:28:08.185437 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.185416 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1accdaf-e3bd-4260-a715-737e35a214c1-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl\" (UID: \"e1accdaf-e3bd-4260-a715-737e35a214c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" Jul 10 22:28:08.185437 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.185429 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1accdaf-e3bd-4260-a715-737e35a214c1-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl\" (UID: \"e1accdaf-e3bd-4260-a715-737e35a214c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" Jul 10 22:28:08.193544 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.193521 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmzdg\" (UniqueName: \"kubernetes.io/projected/e1accdaf-e3bd-4260-a715-737e35a214c1-kube-api-access-vmzdg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl\" (UID: \"e1accdaf-e3bd-4260-a715-737e35a214c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" Jul 10 22:28:08.298754 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.298661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" Jul 10 22:28:08.428503 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.428368 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl"] Jul 10 22:28:08.430750 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:28:08.430721 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1accdaf_e3bd_4260_a715_737e35a214c1.slice/crio-4e2a5a7102cc45a1981a3ec4c6f9c0e7a8ad33279974325a3cda5533cff5dfc1 WatchSource:0}: Error finding container 4e2a5a7102cc45a1981a3ec4c6f9c0e7a8ad33279974325a3cda5533cff5dfc1: Status 404 returned error can't find the container with id 4e2a5a7102cc45a1981a3ec4c6f9c0e7a8ad33279974325a3cda5533cff5dfc1 Jul 10 22:28:08.536857 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:08.536786 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1accdaf_e3bd_4260_a715_737e35a214c1.slice/crio-6b58c047eb2afdfbe25e33bd925ffdb5ec590b75263584c13d7794d47b35a310.scope\": RecentStats: unable to find data in memory cache]" Jul 10 22:28:08.574414 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.574335 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2"] Jul 10 22:28:08.577540 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.577523 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" Jul 10 22:28:08.586456 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.586426 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2"] Jul 10 22:28:08.688631 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.688590 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2\" (UID: \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" Jul 10 22:28:08.688844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.688652 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjbpt\" (UniqueName: \"kubernetes.io/projected/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-kube-api-access-gjbpt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2\" (UID: \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" Jul 10 22:28:08.688844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.688690 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2\" (UID: \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" Jul 10 22:28:08.789437 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.789398 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2\" (UID: \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" Jul 10 22:28:08.789610 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.789478 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2\" (UID: \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" Jul 10 22:28:08.789610 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.789526 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjbpt\" (UniqueName: \"kubernetes.io/projected/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-kube-api-access-gjbpt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2\" (UID: \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" Jul 10 22:28:08.789769 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.789750 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2\" (UID: \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" Jul 10 22:28:08.789855 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.789837 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2\" (UID: \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" Jul 10 22:28:08.797057 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.797030 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjbpt\" (UniqueName: \"kubernetes.io/projected/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-kube-api-access-gjbpt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2\" (UID: \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" Jul 10 22:28:08.887310 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:08.887223 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" Jul 10 22:28:09.061268 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.061217 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2"] Jul 10 22:28:09.063473 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:28:09.063436 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c96808_c75e_4edc_9e57_a2aae9c5c0d7.slice/crio-723c4ee05948b7766aaf74c35a74c2e2565e6a30b591f22bccb29219fde90b7e WatchSource:0}: Error finding container 723c4ee05948b7766aaf74c35a74c2e2565e6a30b591f22bccb29219fde90b7e: Status 404 returned error can't find the container with id 723c4ee05948b7766aaf74c35a74c2e2565e6a30b591f22bccb29219fde90b7e Jul 10 22:28:09.148743 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.148714 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" event={"ID":"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7","Type":"ContainerStarted","Data":"723c4ee05948b7766aaf74c35a74c2e2565e6a30b591f22bccb29219fde90b7e"} Jul 10 22:28:09.150565 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.150536 2565 generic.go:356] "Generic (PLEG): container finished" podID="22aca153-5823-443b-bd4e-c0009121fed9" containerID="2a827d808e99b3b5a6c9ef6599d8afd651e1ed586036e24b44c268da43a5422f" exitCode=0 Jul 10 22:28:09.150670 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.150613 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" event={"ID":"22aca153-5823-443b-bd4e-c0009121fed9","Type":"ContainerDied","Data":"2a827d808e99b3b5a6c9ef6599d8afd651e1ed586036e24b44c268da43a5422f"} Jul 10 22:28:09.152111 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.152087 2565 generic.go:356] "Generic (PLEG): container finished" podID="e1accdaf-e3bd-4260-a715-737e35a214c1" containerID="6b58c047eb2afdfbe25e33bd925ffdb5ec590b75263584c13d7794d47b35a310" exitCode=0 Jul 10 22:28:09.152223 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.152160 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" event={"ID":"e1accdaf-e3bd-4260-a715-737e35a214c1","Type":"ContainerDied","Data":"6b58c047eb2afdfbe25e33bd925ffdb5ec590b75263584c13d7794d47b35a310"} Jul 10 22:28:09.152223 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.152193 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" event={"ID":"e1accdaf-e3bd-4260-a715-737e35a214c1","Type":"ContainerStarted","Data":"4e2a5a7102cc45a1981a3ec4c6f9c0e7a8ad33279974325a3cda5533cff5dfc1"} Jul 10 22:28:09.172294 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.172263 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd"] Jul 10 22:28:09.177478 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.177456 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" Jul 10 22:28:09.184459 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.184437 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd"] Jul 10 22:28:09.294258 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.294228 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73e7fae0-1ba2-41e4-a679-85df4e8af497-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd\" (UID: \"73e7fae0-1ba2-41e4-a679-85df4e8af497\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" Jul 10 22:28:09.294369 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.294269 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqk97\" (UniqueName: \"kubernetes.io/projected/73e7fae0-1ba2-41e4-a679-85df4e8af497-kube-api-access-nqk97\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd\" (UID: \"73e7fae0-1ba2-41e4-a679-85df4e8af497\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" Jul 10 22:28:09.294369 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.294302 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73e7fae0-1ba2-41e4-a679-85df4e8af497-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd\" (UID: \"73e7fae0-1ba2-41e4-a679-85df4e8af497\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" Jul 10 22:28:09.395763 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.395682 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73e7fae0-1ba2-41e4-a679-85df4e8af497-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd\" (UID: \"73e7fae0-1ba2-41e4-a679-85df4e8af497\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" Jul 10 22:28:09.395763 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.395718 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqk97\" (UniqueName: \"kubernetes.io/projected/73e7fae0-1ba2-41e4-a679-85df4e8af497-kube-api-access-nqk97\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd\" (UID: \"73e7fae0-1ba2-41e4-a679-85df4e8af497\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" Jul 10 22:28:09.395763 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.395748 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73e7fae0-1ba2-41e4-a679-85df4e8af497-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd\" (UID: \"73e7fae0-1ba2-41e4-a679-85df4e8af497\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" Jul 10 22:28:09.396073 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.396059 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73e7fae0-1ba2-41e4-a679-85df4e8af497-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd\" (UID: \"73e7fae0-1ba2-41e4-a679-85df4e8af497\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" Jul 10 22:28:09.396116 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.396100 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73e7fae0-1ba2-41e4-a679-85df4e8af497-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd\" (UID: \"73e7fae0-1ba2-41e4-a679-85df4e8af497\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" Jul 10 22:28:09.404417 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.404386 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqk97\" (UniqueName: \"kubernetes.io/projected/73e7fae0-1ba2-41e4-a679-85df4e8af497-kube-api-access-nqk97\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd\" (UID: \"73e7fae0-1ba2-41e4-a679-85df4e8af497\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" Jul 10 22:28:09.495935 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.495894 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" Jul 10 22:28:09.627982 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:09.627957 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd"] Jul 10 22:28:09.629633 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:28:09.629604 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e7fae0_1ba2_41e4_a679_85df4e8af497.slice/crio-cdcc6c79147f5fc7d5d8870db9c5910502e0c418b959520d335a010d6d0a9695 WatchSource:0}: Error finding container cdcc6c79147f5fc7d5d8870db9c5910502e0c418b959520d335a010d6d0a9695: Status 404 returned error can't find the container with id cdcc6c79147f5fc7d5d8870db9c5910502e0c418b959520d335a010d6d0a9695 Jul 10 22:28:10.158959 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:10.158930 2565 generic.go:356] "Generic (PLEG): container finished" podID="e1accdaf-e3bd-4260-a715-737e35a214c1" containerID="d4855387fe8801b1270e373085c72a133e5aeabab0e987552fadd73dd5f04c7b" exitCode=0 Jul 10 22:28:10.159072 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:10.159012 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" event={"ID":"e1accdaf-e3bd-4260-a715-737e35a214c1","Type":"ContainerDied","Data":"d4855387fe8801b1270e373085c72a133e5aeabab0e987552fadd73dd5f04c7b"} Jul 10 22:28:10.160556 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:10.160536 2565 generic.go:356] "Generic (PLEG): container finished" podID="73e7fae0-1ba2-41e4-a679-85df4e8af497" containerID="b2d361fb09c4d37c9801d90e36a5b1a3a337b4154b687c4894113b3da52f6a7d" exitCode=0 Jul 10 22:28:10.160632 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:10.160616 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" event={"ID":"73e7fae0-1ba2-41e4-a679-85df4e8af497","Type":"ContainerDied","Data":"b2d361fb09c4d37c9801d90e36a5b1a3a337b4154b687c4894113b3da52f6a7d"} Jul 10 22:28:10.160681 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:10.160659 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" event={"ID":"73e7fae0-1ba2-41e4-a679-85df4e8af497","Type":"ContainerStarted","Data":"cdcc6c79147f5fc7d5d8870db9c5910502e0c418b959520d335a010d6d0a9695"} Jul 10 22:28:10.162253 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:10.162230 2565 generic.go:356] "Generic (PLEG): container finished" podID="d4c96808-c75e-4edc-9e57-a2aae9c5c0d7" containerID="654ec90b58290413d7b29690057e991b6f53186934f8c848feb45636d7dcf590" exitCode=0 Jul 10 22:28:10.162393 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:10.162314 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" event={"ID":"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7","Type":"ContainerDied","Data":"654ec90b58290413d7b29690057e991b6f53186934f8c848feb45636d7dcf590"} Jul 10 22:28:10.164918 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:10.164887 2565 generic.go:356] "Generic (PLEG): container finished" podID="22aca153-5823-443b-bd4e-c0009121fed9" containerID="e84f7b7ee56861143ccb208b05bbc1e3828d62071128d8ea2cd3f96091a0d0cb" exitCode=0 Jul 10 22:28:10.165008 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:10.164921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" event={"ID":"22aca153-5823-443b-bd4e-c0009121fed9","Type":"ContainerDied","Data":"e84f7b7ee56861143ccb208b05bbc1e3828d62071128d8ea2cd3f96091a0d0cb"} Jul 10 22:28:11.171050 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.170952 2565 generic.go:356] "Generic (PLEG): container finished" podID="73e7fae0-1ba2-41e4-a679-85df4e8af497" containerID="6bc28fefda39fdedefd35908375949b1c1950939367a618e46d6600b20ee753d" exitCode=0 Jul 10 22:28:11.171050 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.171031 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" event={"ID":"73e7fae0-1ba2-41e4-a679-85df4e8af497","Type":"ContainerDied","Data":"6bc28fefda39fdedefd35908375949b1c1950939367a618e46d6600b20ee753d"} Jul 10 22:28:11.172953 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.172929 2565 generic.go:356] "Generic (PLEG): container finished" podID="d4c96808-c75e-4edc-9e57-a2aae9c5c0d7" containerID="bf1e2fa91417875dc31c5b108c00ce894bfc8018fcd14feba860f1f281aa9cdf" exitCode=0 Jul 10 22:28:11.173061 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.172986 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" event={"ID":"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7","Type":"ContainerDied","Data":"bf1e2fa91417875dc31c5b108c00ce894bfc8018fcd14feba860f1f281aa9cdf"} Jul 10 22:28:11.175225 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.175205 2565 generic.go:356] "Generic (PLEG): container finished" podID="e1accdaf-e3bd-4260-a715-737e35a214c1" containerID="7d38ab2ef14685ebce89a70e677f4bf85b646bc98df6f9c9ae8a27499d039369" exitCode=0 Jul 10 22:28:11.175313 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.175274 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" event={"ID":"e1accdaf-e3bd-4260-a715-737e35a214c1","Type":"ContainerDied","Data":"7d38ab2ef14685ebce89a70e677f4bf85b646bc98df6f9c9ae8a27499d039369"} Jul 10 22:28:11.311427 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.311393 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" Jul 10 22:28:11.412778 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.412742 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4pfp\" (UniqueName: \"kubernetes.io/projected/22aca153-5823-443b-bd4e-c0009121fed9-kube-api-access-r4pfp\") pod \"22aca153-5823-443b-bd4e-c0009121fed9\" (UID: \"22aca153-5823-443b-bd4e-c0009121fed9\") " Jul 10 22:28:11.412778 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.412778 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22aca153-5823-443b-bd4e-c0009121fed9-bundle\") pod \"22aca153-5823-443b-bd4e-c0009121fed9\" (UID: \"22aca153-5823-443b-bd4e-c0009121fed9\") " Jul 10 22:28:11.413020 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.412825 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22aca153-5823-443b-bd4e-c0009121fed9-util\") pod \"22aca153-5823-443b-bd4e-c0009121fed9\" (UID: \"22aca153-5823-443b-bd4e-c0009121fed9\") " Jul 10 22:28:11.413358 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.413329 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22aca153-5823-443b-bd4e-c0009121fed9-bundle" (OuterVolumeSpecName: "bundle") pod "22aca153-5823-443b-bd4e-c0009121fed9" (UID: "22aca153-5823-443b-bd4e-c0009121fed9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:28:11.414928 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.414907 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22aca153-5823-443b-bd4e-c0009121fed9-kube-api-access-r4pfp" (OuterVolumeSpecName: "kube-api-access-r4pfp") pod "22aca153-5823-443b-bd4e-c0009121fed9" (UID: "22aca153-5823-443b-bd4e-c0009121fed9"). InnerVolumeSpecName "kube-api-access-r4pfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:28:11.418047 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.418023 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22aca153-5823-443b-bd4e-c0009121fed9-util" (OuterVolumeSpecName: "util") pod "22aca153-5823-443b-bd4e-c0009121fed9" (UID: "22aca153-5823-443b-bd4e-c0009121fed9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:28:11.514125 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.514095 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4pfp\" (UniqueName: \"kubernetes.io/projected/22aca153-5823-443b-bd4e-c0009121fed9-kube-api-access-r4pfp\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:11.514125 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.514122 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22aca153-5823-443b-bd4e-c0009121fed9-bundle\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:11.514125 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:11.514131 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22aca153-5823-443b-bd4e-c0009121fed9-util\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:12.182958 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.182924 2565 generic.go:356] "Generic (PLEG): container finished" podID="73e7fae0-1ba2-41e4-a679-85df4e8af497" containerID="d123a84669aec6e062abb397d17064ef50aeecf52793e1bbcc9bbc5e5fa2d007" exitCode=0 Jul 10 22:28:12.183357 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.183021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" event={"ID":"73e7fae0-1ba2-41e4-a679-85df4e8af497","Type":"ContainerDied","Data":"d123a84669aec6e062abb397d17064ef50aeecf52793e1bbcc9bbc5e5fa2d007"} Jul 10 22:28:12.185013 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.184988 2565 generic.go:356] "Generic (PLEG): container finished" podID="d4c96808-c75e-4edc-9e57-a2aae9c5c0d7" containerID="44426590fc5263a624d7bb6645cffb481b23c3b9673262da0e09e63311bffb87" exitCode=0 Jul 10 22:28:12.185117 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.185069 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" event={"ID":"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7","Type":"ContainerDied","Data":"44426590fc5263a624d7bb6645cffb481b23c3b9673262da0e09e63311bffb87"} Jul 10 22:28:12.186740 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.186714 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" event={"ID":"22aca153-5823-443b-bd4e-c0009121fed9","Type":"ContainerDied","Data":"82daad859577159fe42af2c77be577f7a0542143e7ed4dc8e6be3e21654eaa3e"} Jul 10 22:28:12.186855 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.186741 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w" Jul 10 22:28:12.186855 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.186754 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82daad859577159fe42af2c77be577f7a0542143e7ed4dc8e6be3e21654eaa3e" Jul 10 22:28:12.317191 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.317166 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" Jul 10 22:28:12.421059 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.421023 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1accdaf-e3bd-4260-a715-737e35a214c1-util\") pod \"e1accdaf-e3bd-4260-a715-737e35a214c1\" (UID: \"e1accdaf-e3bd-4260-a715-737e35a214c1\") " Jul 10 22:28:12.421227 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.421079 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1accdaf-e3bd-4260-a715-737e35a214c1-bundle\") pod \"e1accdaf-e3bd-4260-a715-737e35a214c1\" (UID: \"e1accdaf-e3bd-4260-a715-737e35a214c1\") " Jul 10 22:28:12.421227 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.421116 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmzdg\" (UniqueName: \"kubernetes.io/projected/e1accdaf-e3bd-4260-a715-737e35a214c1-kube-api-access-vmzdg\") pod \"e1accdaf-e3bd-4260-a715-737e35a214c1\" (UID: \"e1accdaf-e3bd-4260-a715-737e35a214c1\") " Jul 10 22:28:12.421716 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.421687 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1accdaf-e3bd-4260-a715-737e35a214c1-bundle" (OuterVolumeSpecName: "bundle") pod "e1accdaf-e3bd-4260-a715-737e35a214c1" (UID: "e1accdaf-e3bd-4260-a715-737e35a214c1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:28:12.423193 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.423173 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1accdaf-e3bd-4260-a715-737e35a214c1-kube-api-access-vmzdg" (OuterVolumeSpecName: "kube-api-access-vmzdg") pod "e1accdaf-e3bd-4260-a715-737e35a214c1" (UID: "e1accdaf-e3bd-4260-a715-737e35a214c1"). InnerVolumeSpecName "kube-api-access-vmzdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:28:12.426315 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.426275 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1accdaf-e3bd-4260-a715-737e35a214c1-util" (OuterVolumeSpecName: "util") pod "e1accdaf-e3bd-4260-a715-737e35a214c1" (UID: "e1accdaf-e3bd-4260-a715-737e35a214c1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:28:12.521731 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.521695 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1accdaf-e3bd-4260-a715-737e35a214c1-util\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:12.521731 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.521725 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1accdaf-e3bd-4260-a715-737e35a214c1-bundle\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:12.521731 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:12.521733 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmzdg\" (UniqueName: \"kubernetes.io/projected/e1accdaf-e3bd-4260-a715-737e35a214c1-kube-api-access-vmzdg\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:13.193229 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.193196 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" Jul 10 22:28:13.193667 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.193198 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl" event={"ID":"e1accdaf-e3bd-4260-a715-737e35a214c1","Type":"ContainerDied","Data":"4e2a5a7102cc45a1981a3ec4c6f9c0e7a8ad33279974325a3cda5533cff5dfc1"} Jul 10 22:28:13.193667 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.193311 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e2a5a7102cc45a1981a3ec4c6f9c0e7a8ad33279974325a3cda5533cff5dfc1" Jul 10 22:28:13.317346 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.317322 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" Jul 10 22:28:13.348101 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.348076 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" Jul 10 22:28:13.430222 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.430192 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73e7fae0-1ba2-41e4-a679-85df4e8af497-util\") pod \"73e7fae0-1ba2-41e4-a679-85df4e8af497\" (UID: \"73e7fae0-1ba2-41e4-a679-85df4e8af497\") " Jul 10 22:28:13.430405 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.430262 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqk97\" (UniqueName: \"kubernetes.io/projected/73e7fae0-1ba2-41e4-a679-85df4e8af497-kube-api-access-nqk97\") pod \"73e7fae0-1ba2-41e4-a679-85df4e8af497\" (UID: \"73e7fae0-1ba2-41e4-a679-85df4e8af497\") " Jul 10 22:28:13.430405 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.430290 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73e7fae0-1ba2-41e4-a679-85df4e8af497-bundle\") pod \"73e7fae0-1ba2-41e4-a679-85df4e8af497\" (UID: \"73e7fae0-1ba2-41e4-a679-85df4e8af497\") " Jul 10 22:28:13.430520 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.430440 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjbpt\" (UniqueName: \"kubernetes.io/projected/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-kube-api-access-gjbpt\") pod \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\" (UID: \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\") " Jul 10 22:28:13.430520 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.430479 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-util\") pod \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\" (UID: \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\") " Jul 10 22:28:13.430520 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.430507 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-bundle\") pod \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\" (UID: \"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7\") " Jul 10 22:28:13.431023 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.430995 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e7fae0-1ba2-41e4-a679-85df4e8af497-bundle" (OuterVolumeSpecName: "bundle") pod "73e7fae0-1ba2-41e4-a679-85df4e8af497" (UID: "73e7fae0-1ba2-41e4-a679-85df4e8af497"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:28:13.431133 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.431071 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-bundle" (OuterVolumeSpecName: "bundle") pod "d4c96808-c75e-4edc-9e57-a2aae9c5c0d7" (UID: "d4c96808-c75e-4edc-9e57-a2aae9c5c0d7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:28:13.432421 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.432403 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e7fae0-1ba2-41e4-a679-85df4e8af497-kube-api-access-nqk97" (OuterVolumeSpecName: "kube-api-access-nqk97") pod "73e7fae0-1ba2-41e4-a679-85df4e8af497" (UID: "73e7fae0-1ba2-41e4-a679-85df4e8af497"). InnerVolumeSpecName "kube-api-access-nqk97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:28:13.432889 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.432868 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-kube-api-access-gjbpt" (OuterVolumeSpecName: "kube-api-access-gjbpt") pod "d4c96808-c75e-4edc-9e57-a2aae9c5c0d7" (UID: "d4c96808-c75e-4edc-9e57-a2aae9c5c0d7"). InnerVolumeSpecName "kube-api-access-gjbpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:28:13.436842 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.436821 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e7fae0-1ba2-41e4-a679-85df4e8af497-util" (OuterVolumeSpecName: "util") pod "73e7fae0-1ba2-41e4-a679-85df4e8af497" (UID: "73e7fae0-1ba2-41e4-a679-85df4e8af497"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:28:13.438741 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.438717 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-util" (OuterVolumeSpecName: "util") pod "d4c96808-c75e-4edc-9e57-a2aae9c5c0d7" (UID: "d4c96808-c75e-4edc-9e57-a2aae9c5c0d7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:28:13.532159 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.532120 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-bundle\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:13.532159 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.532152 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73e7fae0-1ba2-41e4-a679-85df4e8af497-util\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:13.532159 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.532162 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqk97\" (UniqueName: \"kubernetes.io/projected/73e7fae0-1ba2-41e4-a679-85df4e8af497-kube-api-access-nqk97\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:13.532387 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.532173 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73e7fae0-1ba2-41e4-a679-85df4e8af497-bundle\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:13.532387 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.532182 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjbpt\" (UniqueName: \"kubernetes.io/projected/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-kube-api-access-gjbpt\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:13.532387 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:13.532190 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4c96808-c75e-4edc-9e57-a2aae9c5c0d7-util\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:14.199642 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:14.199611 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" Jul 10 22:28:14.199642 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:14.199633 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2" event={"ID":"d4c96808-c75e-4edc-9e57-a2aae9c5c0d7","Type":"ContainerDied","Data":"723c4ee05948b7766aaf74c35a74c2e2565e6a30b591f22bccb29219fde90b7e"} Jul 10 22:28:14.200125 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:14.199714 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="723c4ee05948b7766aaf74c35a74c2e2565e6a30b591f22bccb29219fde90b7e" Jul 10 22:28:14.201760 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:14.201735 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" event={"ID":"73e7fae0-1ba2-41e4-a679-85df4e8af497","Type":"ContainerDied","Data":"cdcc6c79147f5fc7d5d8870db9c5910502e0c418b959520d335a010d6d0a9695"} Jul 10 22:28:14.201910 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:14.201779 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdcc6c79147f5fc7d5d8870db9c5910502e0c418b959520d335a010d6d0a9695" Jul 10 22:28:14.201910 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:14.201779 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd" Jul 10 22:28:15.452807 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:15.452768 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:17.897221 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:17.897194 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:20.360853 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.360819 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6"] Jul 10 22:28:20.361333 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361278 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4c96808-c75e-4edc-9e57-a2aae9c5c0d7" containerName="extract" Jul 10 22:28:20.361333 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361298 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4c96808-c75e-4edc-9e57-a2aae9c5c0d7" containerName="extract" Jul 10 22:28:20.361333 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361310 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1accdaf-e3bd-4260-a715-737e35a214c1" containerName="util" Jul 10 22:28:20.361333 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361318 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1accdaf-e3bd-4260-a715-737e35a214c1" containerName="util" Jul 10 22:28:20.361333 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361332 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4c96808-c75e-4edc-9e57-a2aae9c5c0d7" containerName="pull" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361340 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4c96808-c75e-4edc-9e57-a2aae9c5c0d7" containerName="pull" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361353 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4c96808-c75e-4edc-9e57-a2aae9c5c0d7" containerName="util" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361362 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4c96808-c75e-4edc-9e57-a2aae9c5c0d7" containerName="util" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361373 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22aca153-5823-443b-bd4e-c0009121fed9" containerName="util" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361382 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22aca153-5823-443b-bd4e-c0009121fed9" containerName="util" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361400 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e7fae0-1ba2-41e4-a679-85df4e8af497" containerName="extract" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361408 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e7fae0-1ba2-41e4-a679-85df4e8af497" containerName="extract" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361418 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e7fae0-1ba2-41e4-a679-85df4e8af497" containerName="util" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361426 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e7fae0-1ba2-41e4-a679-85df4e8af497" containerName="util" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361438 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22aca153-5823-443b-bd4e-c0009121fed9" containerName="extract" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361446 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22aca153-5823-443b-bd4e-c0009121fed9" containerName="extract" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361455 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1accdaf-e3bd-4260-a715-737e35a214c1" containerName="extract" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361463 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1accdaf-e3bd-4260-a715-737e35a214c1" containerName="extract" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361475 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1accdaf-e3bd-4260-a715-737e35a214c1" containerName="pull" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361483 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1accdaf-e3bd-4260-a715-737e35a214c1" containerName="pull" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361494 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22aca153-5823-443b-bd4e-c0009121fed9" containerName="pull" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361501 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22aca153-5823-443b-bd4e-c0009121fed9" containerName="pull" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361511 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e7fae0-1ba2-41e4-a679-85df4e8af497" containerName="pull" Jul 10 22:28:20.361584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361519 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e7fae0-1ba2-41e4-a679-85df4e8af497" containerName="pull" Jul 10 22:28:20.362482 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361593 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e7fae0-1ba2-41e4-a679-85df4e8af497" containerName="extract" Jul 10 22:28:20.362482 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361605 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22aca153-5823-443b-bd4e-c0009121fed9" containerName="extract" Jul 10 22:28:20.362482 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361614 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4c96808-c75e-4edc-9e57-a2aae9c5c0d7" containerName="extract" Jul 10 22:28:20.362482 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.361624 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1accdaf-e3bd-4260-a715-737e35a214c1" containerName="extract" Jul 10 22:28:20.366068 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.366049 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:20.368970 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.368954 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-c7znz\"" Jul 10 22:28:20.379532 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.379509 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6"] Jul 10 22:28:20.488376 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.488331 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea5f39d9-0d24-4448-9b13-b8d8e3bd1660-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" (UID: \"ea5f39d9-0d24-4448-9b13-b8d8e3bd1660\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:20.488376 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.488383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kps2t\" (UniqueName: \"kubernetes.io/projected/ea5f39d9-0d24-4448-9b13-b8d8e3bd1660-kube-api-access-kps2t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" (UID: \"ea5f39d9-0d24-4448-9b13-b8d8e3bd1660\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:20.589628 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.589583 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea5f39d9-0d24-4448-9b13-b8d8e3bd1660-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" (UID: \"ea5f39d9-0d24-4448-9b13-b8d8e3bd1660\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:20.589860 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.589642 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kps2t\" (UniqueName: \"kubernetes.io/projected/ea5f39d9-0d24-4448-9b13-b8d8e3bd1660-kube-api-access-kps2t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" (UID: \"ea5f39d9-0d24-4448-9b13-b8d8e3bd1660\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:20.589987 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.589964 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea5f39d9-0d24-4448-9b13-b8d8e3bd1660-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" (UID: \"ea5f39d9-0d24-4448-9b13-b8d8e3bd1660\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:20.599718 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.599689 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kps2t\" (UniqueName: \"kubernetes.io/projected/ea5f39d9-0d24-4448-9b13-b8d8e3bd1660-kube-api-access-kps2t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" (UID: \"ea5f39d9-0d24-4448-9b13-b8d8e3bd1660\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:20.675754 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.675663 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:20.815334 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:20.815308 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6"] Jul 10 22:28:20.817347 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:28:20.817323 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5f39d9_0d24_4448_9b13_b8d8e3bd1660.slice/crio-3d2580c2fdf30486c28faa5abf942ecb5dcc3ad2eb737ca8fd0af31834cc5268 WatchSource:0}: Error finding container 3d2580c2fdf30486c28faa5abf942ecb5dcc3ad2eb737ca8fd0af31834cc5268: Status 404 returned error can't find the container with id 3d2580c2fdf30486c28faa5abf942ecb5dcc3ad2eb737ca8fd0af31834cc5268 Jul 10 22:28:21.236365 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:21.236326 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" event={"ID":"ea5f39d9-0d24-4448-9b13-b8d8e3bd1660","Type":"ContainerStarted","Data":"3d2580c2fdf30486c28faa5abf942ecb5dcc3ad2eb737ca8fd0af31834cc5268"} Jul 10 22:28:21.318142 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:21.318110 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:22.962464 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:22.962414 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:23.266089 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:23.266047 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-d9pwm"] Jul 10 22:28:23.269464 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:23.269425 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d9pwm" Jul 10 22:28:23.272105 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:23.272066 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 22:28:23.272230 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:23.272137 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-57v5t\"" Jul 10 22:28:23.287558 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:23.287526 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-d9pwm"] Jul 10 22:28:23.416651 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:23.416582 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t5d9\" (UniqueName: \"kubernetes.io/projected/ad0186f0-e17b-4677-85a6-feb5853a3587-kube-api-access-9t5d9\") pod \"dns-operator-controller-manager-864dd78c4c-d9pwm\" (UID: \"ad0186f0-e17b-4677-85a6-feb5853a3587\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d9pwm" Jul 10 22:28:23.518093 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:23.518006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t5d9\" (UniqueName: \"kubernetes.io/projected/ad0186f0-e17b-4677-85a6-feb5853a3587-kube-api-access-9t5d9\") pod \"dns-operator-controller-manager-864dd78c4c-d9pwm\" (UID: \"ad0186f0-e17b-4677-85a6-feb5853a3587\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d9pwm" Jul 10 22:28:23.530862 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:23.530789 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t5d9\" (UniqueName: \"kubernetes.io/projected/ad0186f0-e17b-4677-85a6-feb5853a3587-kube-api-access-9t5d9\") pod \"dns-operator-controller-manager-864dd78c4c-d9pwm\" (UID: \"ad0186f0-e17b-4677-85a6-feb5853a3587\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d9pwm" Jul 10 22:28:23.583192 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:23.583152 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d9pwm" Jul 10 22:28:23.722310 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:23.722279 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-d9pwm"] Jul 10 22:28:23.724882 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:28:23.724853 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0186f0_e17b_4677_85a6_feb5853a3587.slice/crio-43553f98c2e902f3f241334d758da55c47b44f7b3f179e72a35bcec948f02c13 WatchSource:0}: Error finding container 43553f98c2e902f3f241334d758da55c47b44f7b3f179e72a35bcec948f02c13: Status 404 returned error can't find the container with id 43553f98c2e902f3f241334d758da55c47b44f7b3f179e72a35bcec948f02c13 Jul 10 22:28:23.878227 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:23.878137 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:24.252457 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:24.252421 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d9pwm" event={"ID":"ad0186f0-e17b-4677-85a6-feb5853a3587","Type":"ContainerStarted","Data":"43553f98c2e902f3f241334d758da55c47b44f7b3f179e72a35bcec948f02c13"} Jul 10 22:28:24.643888 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:24.643782 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:27.267923 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.267880 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" event={"ID":"ea5f39d9-0d24-4448-9b13-b8d8e3bd1660","Type":"ContainerStarted","Data":"71e5c63f880a2530ddc560888635b51490dcbe9c690e876b23683bd2f4034f56"} Jul 10 22:28:27.268341 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.268015 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:27.269432 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.269409 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d9pwm" event={"ID":"ad0186f0-e17b-4677-85a6-feb5853a3587","Type":"ContainerStarted","Data":"6388c7fb8852a3e27e350d32750ccdfe32d7326b9446ff362be965bcbae88870"} Jul 10 22:28:27.269511 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.269452 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d9pwm" Jul 10 22:28:27.287209 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.287160 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" podStartSLOduration=1.233681329 podStartE2EDuration="7.287145231s" totalImagesPullingTime="6.053463902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:28:27.285359391 +0000 UTC m=+371.118243774" watchObservedRunningTime="2026-07-10 22:28:27.287145231 +0000 UTC m=+371.120029617" Jul 10 22:28:27.307623 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.307584 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d9pwm" podStartSLOduration=1.168802013 podStartE2EDuration="4.307570802s" totalImagesPullingTime="3.138768789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:28:27.30632852 +0000 UTC m=+371.139212904" watchObservedRunningTime="2026-07-10 22:28:27.307570802 +0000 UTC m=+371.140455186" Jul 10 22:28:27.453032 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.452998 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2hpnt"] Jul 10 22:28:27.456281 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.456266 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2hpnt" Jul 10 22:28:27.458444 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.458427 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-srz4j\"" Jul 10 22:28:27.466763 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.466740 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2hpnt"] Jul 10 22:28:27.555495 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.555403 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9769t\" (UniqueName: \"kubernetes.io/projected/f7e899dc-a39f-418e-8ebe-e6fbb9967c09-kube-api-access-9769t\") pod \"authorino-operator-75cc8d5cb7-2hpnt\" (UID: \"f7e899dc-a39f-418e-8ebe-e6fbb9967c09\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2hpnt" Jul 10 22:28:27.656502 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.656456 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9769t\" (UniqueName: \"kubernetes.io/projected/f7e899dc-a39f-418e-8ebe-e6fbb9967c09-kube-api-access-9769t\") pod \"authorino-operator-75cc8d5cb7-2hpnt\" (UID: \"f7e899dc-a39f-418e-8ebe-e6fbb9967c09\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2hpnt" Jul 10 22:28:27.671186 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.671150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9769t\" (UniqueName: \"kubernetes.io/projected/f7e899dc-a39f-418e-8ebe-e6fbb9967c09-kube-api-access-9769t\") pod \"authorino-operator-75cc8d5cb7-2hpnt\" (UID: \"f7e899dc-a39f-418e-8ebe-e6fbb9967c09\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2hpnt" Jul 10 22:28:27.766604 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.766567 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2hpnt" Jul 10 22:28:27.904892 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:27.902220 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2hpnt"] Jul 10 22:28:28.277921 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:28.277884 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2hpnt" event={"ID":"f7e899dc-a39f-418e-8ebe-e6fbb9967c09","Type":"ContainerStarted","Data":"42bc76f49a24a799eee441a4c6f1a3d8302944e8ddcf813a49cb84c1f8a09578"} Jul 10 22:28:28.462476 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:28.462440 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:29.951159 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:29.951121 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm"] Jul 10 22:28:29.954908 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:29.954884 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" Jul 10 22:28:29.957063 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:29.957030 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 22:28:29.957063 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:29.957048 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 22:28:29.957222 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:29.957115 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tp8vl\"" Jul 10 22:28:29.963065 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:29.963038 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:29.965371 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:29.965320 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm"] Jul 10 22:28:30.056472 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.056438 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:30.076584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.076549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6436dd9-f13f-4e4b-abf0-9fa23884e7ea-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zxzzm\" (UID: \"b6436dd9-f13f-4e4b-abf0-9fa23884e7ea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" Jul 10 22:28:30.076859 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.076601 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5d7j\" (UniqueName: \"kubernetes.io/projected/b6436dd9-f13f-4e4b-abf0-9fa23884e7ea-kube-api-access-n5d7j\") pod \"kuadrant-console-plugin-66b5774985-zxzzm\" (UID: \"b6436dd9-f13f-4e4b-abf0-9fa23884e7ea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" Jul 10 22:28:30.076859 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.076702 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6436dd9-f13f-4e4b-abf0-9fa23884e7ea-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zxzzm\" (UID: \"b6436dd9-f13f-4e4b-abf0-9fa23884e7ea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" Jul 10 22:28:30.177496 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.177470 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6436dd9-f13f-4e4b-abf0-9fa23884e7ea-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zxzzm\" (UID: \"b6436dd9-f13f-4e4b-abf0-9fa23884e7ea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" Jul 10 22:28:30.177581 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.177521 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6436dd9-f13f-4e4b-abf0-9fa23884e7ea-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zxzzm\" (UID: \"b6436dd9-f13f-4e4b-abf0-9fa23884e7ea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" Jul 10 22:28:30.177700 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.177676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5d7j\" (UniqueName: \"kubernetes.io/projected/b6436dd9-f13f-4e4b-abf0-9fa23884e7ea-kube-api-access-n5d7j\") pod \"kuadrant-console-plugin-66b5774985-zxzzm\" (UID: \"b6436dd9-f13f-4e4b-abf0-9fa23884e7ea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" Jul 10 22:28:30.178175 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.178152 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6436dd9-f13f-4e4b-abf0-9fa23884e7ea-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zxzzm\" (UID: \"b6436dd9-f13f-4e4b-abf0-9fa23884e7ea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" Jul 10 22:28:30.180029 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.179952 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6436dd9-f13f-4e4b-abf0-9fa23884e7ea-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zxzzm\" (UID: \"b6436dd9-f13f-4e4b-abf0-9fa23884e7ea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" Jul 10 22:28:30.185421 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.185389 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5d7j\" (UniqueName: \"kubernetes.io/projected/b6436dd9-f13f-4e4b-abf0-9fa23884e7ea-kube-api-access-n5d7j\") pod \"kuadrant-console-plugin-66b5774985-zxzzm\" (UID: \"b6436dd9-f13f-4e4b-abf0-9fa23884e7ea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" Jul 10 22:28:30.267819 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.267768 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" Jul 10 22:28:30.291258 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.291226 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2hpnt" event={"ID":"f7e899dc-a39f-418e-8ebe-e6fbb9967c09","Type":"ContainerStarted","Data":"a5f66fbef9de34cc4163514dd13e6f4ea97508830a35471950c2c38e4162e51d"} Jul 10 22:28:30.291401 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.291338 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2hpnt" Jul 10 22:28:30.307689 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.307635 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2hpnt" podStartSLOduration=1.047369816 podStartE2EDuration="3.307617101s" totalImagesPullingTime="2.260247285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:28:30.306906405 +0000 UTC m=+374.139790815" watchObservedRunningTime="2026-07-10 22:28:30.307617101 +0000 UTC m=+374.140501484" Jul 10 22:28:30.399995 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.399969 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm"] Jul 10 22:28:30.445295 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.445256 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84"] Jul 10 22:28:30.449976 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.449959 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:30.452161 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.452144 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fm55j\"" Jul 10 22:28:30.457672 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.457640 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84"] Jul 10 22:28:30.580357 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.580321 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf8kw\" (UniqueName: \"kubernetes.io/projected/60d1a470-ca0b-4320-b949-7ec0613cbd0d-kube-api-access-lf8kw\") pod \"limitador-operator-controller-manager-859d7b78bd-6wl84\" (UID: \"60d1a470-ca0b-4320-b949-7ec0613cbd0d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:30.681250 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.681213 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf8kw\" (UniqueName: \"kubernetes.io/projected/60d1a470-ca0b-4320-b949-7ec0613cbd0d-kube-api-access-lf8kw\") pod \"limitador-operator-controller-manager-859d7b78bd-6wl84\" (UID: \"60d1a470-ca0b-4320-b949-7ec0613cbd0d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:30.689569 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.689545 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf8kw\" (UniqueName: \"kubernetes.io/projected/60d1a470-ca0b-4320-b949-7ec0613cbd0d-kube-api-access-lf8kw\") pod \"limitador-operator-controller-manager-859d7b78bd-6wl84\" (UID: \"60d1a470-ca0b-4320-b949-7ec0613cbd0d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:30.761552 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.761517 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:30.905044 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:30.905015 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84"] Jul 10 22:28:30.907049 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:28:30.907017 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d1a470_ca0b_4320_b949_7ec0613cbd0d.slice/crio-ad617bcb6de74435bd71d2504ac1ae5cc5a149e30795b80d09eb9703fe1d4394 WatchSource:0}: Error finding container ad617bcb6de74435bd71d2504ac1ae5cc5a149e30795b80d09eb9703fe1d4394: Status 404 returned error can't find the container with id ad617bcb6de74435bd71d2504ac1ae5cc5a149e30795b80d09eb9703fe1d4394 Jul 10 22:28:31.297860 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:31.297828 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" event={"ID":"60d1a470-ca0b-4320-b949-7ec0613cbd0d","Type":"ContainerStarted","Data":"ad617bcb6de74435bd71d2504ac1ae5cc5a149e30795b80d09eb9703fe1d4394"} Jul 10 22:28:31.299368 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:31.299339 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" event={"ID":"b6436dd9-f13f-4e4b-abf0-9fa23884e7ea","Type":"ContainerStarted","Data":"002e4f5e5b0a807701024cc7af24290f7e37b934655faceeb9aacdce7f5a01cb"} Jul 10 22:28:31.971968 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:31.971935 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:32.072068 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:32.072022 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:33.054813 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:33.054654 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:33.116355 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:33.116310 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:33.312188 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:33.312101 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" event={"ID":"60d1a470-ca0b-4320-b949-7ec0613cbd0d","Type":"ContainerStarted","Data":"9afc9654afbde2e90e0ac3f9ba2b8b65f2c13170682730bed6603622707656b7"} Jul 10 22:28:33.312372 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:33.312239 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:33.330920 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:33.330858 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" podStartSLOduration=1.568240196 podStartE2EDuration="3.330838548s" totalImagesPullingTime="1.762598352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:28:33.329150314 +0000 UTC m=+377.162034698" watchObservedRunningTime="2026-07-10 22:28:33.330838548 +0000 UTC m=+377.163722932" Jul 10 22:28:38.280391 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:38.280354 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:38.280950 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:38.280443 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d9pwm" Jul 10 22:28:40.235221 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.235181 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6"] Jul 10 22:28:40.235758 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.235466 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" containerName="manager" containerID="cri-o://71e5c63f880a2530ddc560888635b51490dcbe9c690e876b23683bd2f4034f56" gracePeriod=2 Jul 10 22:28:40.249423 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.249388 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6"] Jul 10 22:28:40.254082 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:40.254013 2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ea5f39d9-0d24-4448-9b13-b8d8e3bd1660\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\": pods \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:40.273325 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.273275 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84"] Jul 10 22:28:40.273624 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.273572 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" containerName="manager" containerID="cri-o://9afc9654afbde2e90e0ac3f9ba2b8b65f2c13170682730bed6603622707656b7" gracePeriod=2 Jul 10 22:28:40.276595 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.276564 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:40.277494 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.277454 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n"] Jul 10 22:28:40.278015 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.277949 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" containerName="manager" Jul 10 22:28:40.278015 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.277972 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" containerName="manager" Jul 10 22:28:40.278177 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.278095 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" containerName="manager" Jul 10 22:28:40.280698 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.280672 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n" Jul 10 22:28:40.282437 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.282364 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84"] Jul 10 22:28:40.288964 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:40.287306 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:40.290501 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:40.290463 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6wl84\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:40.295363 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:40.295324 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:40.312585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.312532 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n"] Jul 10 22:28:40.315377 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.315343 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzp7b"] Jul 10 22:28:40.316039 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.316020 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" containerName="manager" Jul 10 22:28:40.316168 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.316148 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" containerName="manager" Jul 10 22:28:40.316304 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.316289 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" containerName="manager" Jul 10 22:28:40.318435 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.318412 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzp7b" Jul 10 22:28:40.328889 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:40.328843 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:40.332093 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.331739 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzp7b"] Jul 10 22:28:40.347585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.347554 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n"] Jul 10 22:28:40.352504 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:40.352467 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-f55mc], unattached volumes=[], failed to process volumes=[extensions-socket-volume kube-api-access-f55mc]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n" podUID="17947eb5-8801-4322-997a-82a0dd4b1d41" Jul 10 22:28:40.358263 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:40.358224 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6wl84\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:40.370944 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.370910 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17947eb5-8801-4322-997a-82a0dd4b1d41-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8th6n\" (UID: \"17947eb5-8801-4322-997a-82a0dd4b1d41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n" Jul 10 22:28:40.371099 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.370968 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55mc\" (UniqueName: \"kubernetes.io/projected/17947eb5-8801-4322-997a-82a0dd4b1d41-kube-api-access-f55mc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8th6n\" (UID: \"17947eb5-8801-4322-997a-82a0dd4b1d41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n" Jul 10 22:28:40.472153 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.472107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17947eb5-8801-4322-997a-82a0dd4b1d41-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8th6n\" (UID: \"17947eb5-8801-4322-997a-82a0dd4b1d41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n" Jul 10 22:28:40.472153 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.472159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcspz\" (UniqueName: \"kubernetes.io/projected/2ffca01f-3f3a-496c-93ac-0d46d9f27915-kube-api-access-mcspz\") pod \"limitador-operator-controller-manager-859d7b78bd-pzp7b\" (UID: \"2ffca01f-3f3a-496c-93ac-0d46d9f27915\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzp7b" Jul 10 22:28:40.472436 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.472190 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55mc\" (UniqueName: \"kubernetes.io/projected/17947eb5-8801-4322-997a-82a0dd4b1d41-kube-api-access-f55mc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8th6n\" (UID: \"17947eb5-8801-4322-997a-82a0dd4b1d41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n" Jul 10 22:28:40.472575 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.472549 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17947eb5-8801-4322-997a-82a0dd4b1d41-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8th6n\" (UID: \"17947eb5-8801-4322-997a-82a0dd4b1d41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n" Jul 10 22:28:40.482275 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.482248 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55mc\" (UniqueName: \"kubernetes.io/projected/17947eb5-8801-4322-997a-82a0dd4b1d41-kube-api-access-f55mc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8th6n\" (UID: \"17947eb5-8801-4322-997a-82a0dd4b1d41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n" Jul 10 22:28:40.573505 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.573407 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcspz\" (UniqueName: \"kubernetes.io/projected/2ffca01f-3f3a-496c-93ac-0d46d9f27915-kube-api-access-mcspz\") pod \"limitador-operator-controller-manager-859d7b78bd-pzp7b\" (UID: \"2ffca01f-3f3a-496c-93ac-0d46d9f27915\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzp7b" Jul 10 22:28:40.582631 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.582595 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcspz\" (UniqueName: \"kubernetes.io/projected/2ffca01f-3f3a-496c-93ac-0d46d9f27915-kube-api-access-mcspz\") pod \"limitador-operator-controller-manager-859d7b78bd-pzp7b\" (UID: \"2ffca01f-3f3a-496c-93ac-0d46d9f27915\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzp7b" Jul 10 22:28:40.660845 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:40.660809 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzp7b" Jul 10 22:28:41.270637 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:41.270602 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:41.301981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:41.301947 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2hpnt" Jul 10 22:28:41.320026 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:41.319973 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6wl84\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:41.321652 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:41.321618 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:41.359871 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:41.359828 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:41.360492 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:41.360461 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n" Jul 10 22:28:41.362546 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:41.362511 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:41.376379 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:41.376241 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n" Jul 10 22:28:41.379313 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:41.379273 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6wl84\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:41.380918 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:41.380879 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:41.385435 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:41.385407 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6wl84\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:41.480705 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:41.480667 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f55mc\" (UniqueName: \"kubernetes.io/projected/17947eb5-8801-4322-997a-82a0dd4b1d41-kube-api-access-f55mc\") pod \"17947eb5-8801-4322-997a-82a0dd4b1d41\" (UID: \"17947eb5-8801-4322-997a-82a0dd4b1d41\") " Jul 10 22:28:41.480920 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:41.480728 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17947eb5-8801-4322-997a-82a0dd4b1d41-extensions-socket-volume\") pod \"17947eb5-8801-4322-997a-82a0dd4b1d41\" (UID: \"17947eb5-8801-4322-997a-82a0dd4b1d41\") " Jul 10 22:28:41.481094 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:41.481060 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17947eb5-8801-4322-997a-82a0dd4b1d41-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "17947eb5-8801-4322-997a-82a0dd4b1d41" (UID: "17947eb5-8801-4322-997a-82a0dd4b1d41"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:28:41.483279 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:41.483255 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17947eb5-8801-4322-997a-82a0dd4b1d41-kube-api-access-f55mc" (OuterVolumeSpecName: "kube-api-access-f55mc") pod "17947eb5-8801-4322-997a-82a0dd4b1d41" (UID: "17947eb5-8801-4322-997a-82a0dd4b1d41"). InnerVolumeSpecName "kube-api-access-f55mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:28:41.581873 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:41.581775 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55mc\" (UniqueName: \"kubernetes.io/projected/17947eb5-8801-4322-997a-82a0dd4b1d41-kube-api-access-f55mc\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:41.581873 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:41.581833 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17947eb5-8801-4322-997a-82a0dd4b1d41-extensions-socket-volume\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:42.365705 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:42.365669 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n" Jul 10 22:28:42.404375 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:42.404337 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n"] Jul 10 22:28:42.408142 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:42.408110 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8th6n"] Jul 10 22:28:43.691755 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:43.691721 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17947eb5-8801-4322-997a-82a0dd4b1d41" path="/var/lib/kubelet/pods/17947eb5-8801-4322-997a-82a0dd4b1d41/volumes" Jul 10 22:28:45.486596 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:45.486560 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:45.587659 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:45.587625 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:47.256624 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:47.256587 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:47.356531 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:47.356489 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:47.484153 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:47.484124 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:47.654496 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:47.654416 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:47.715483 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:47.715448 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:51.250921 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:51.250883 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:51.351713 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:51.351673 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:53.310983 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.310956 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:53.317638 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:53.317598 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6wl84\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:53.320453 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.320346 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzp7b"] Jul 10 22:28:53.321512 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:28:53.321486 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffca01f_3f3a_496c_93ac_0d46d9f27915.slice/crio-2f57bbc8cb40717318b7621b2640213eb303fb9ff99af1b9ab783f73fe354c39 WatchSource:0}: Error finding container 2f57bbc8cb40717318b7621b2640213eb303fb9ff99af1b9ab783f73fe354c39: Status 404 returned error can't find the container with id 2f57bbc8cb40717318b7621b2640213eb303fb9ff99af1b9ab783f73fe354c39 Jul 10 22:28:53.323888 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.323863 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:53.326519 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:53.326488 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:53.328834 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:53.328808 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6wl84\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:53.387276 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.387256 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea5f39d9-0d24-4448-9b13-b8d8e3bd1660-extensions-socket-volume\") pod \"ea5f39d9-0d24-4448-9b13-b8d8e3bd1660\" (UID: \"ea5f39d9-0d24-4448-9b13-b8d8e3bd1660\") " Jul 10 22:28:53.387372 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.387296 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf8kw\" (UniqueName: \"kubernetes.io/projected/60d1a470-ca0b-4320-b949-7ec0613cbd0d-kube-api-access-lf8kw\") pod \"60d1a470-ca0b-4320-b949-7ec0613cbd0d\" (UID: \"60d1a470-ca0b-4320-b949-7ec0613cbd0d\") " Jul 10 22:28:53.387431 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.387377 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kps2t\" (UniqueName: \"kubernetes.io/projected/ea5f39d9-0d24-4448-9b13-b8d8e3bd1660-kube-api-access-kps2t\") pod \"ea5f39d9-0d24-4448-9b13-b8d8e3bd1660\" (UID: \"ea5f39d9-0d24-4448-9b13-b8d8e3bd1660\") " Jul 10 22:28:53.387820 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.387775 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5f39d9-0d24-4448-9b13-b8d8e3bd1660-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" (UID: "ea5f39d9-0d24-4448-9b13-b8d8e3bd1660"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:28:53.389348 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.389321 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d1a470-ca0b-4320-b949-7ec0613cbd0d-kube-api-access-lf8kw" (OuterVolumeSpecName: "kube-api-access-lf8kw") pod "60d1a470-ca0b-4320-b949-7ec0613cbd0d" (UID: "60d1a470-ca0b-4320-b949-7ec0613cbd0d"). InnerVolumeSpecName "kube-api-access-lf8kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:28:53.389680 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.389660 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5f39d9-0d24-4448-9b13-b8d8e3bd1660-kube-api-access-kps2t" (OuterVolumeSpecName: "kube-api-access-kps2t") pod "ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" (UID: "ea5f39d9-0d24-4448-9b13-b8d8e3bd1660"). InnerVolumeSpecName "kube-api-access-kps2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:28:53.428453 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.428335 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" event={"ID":"b6436dd9-f13f-4e4b-abf0-9fa23884e7ea","Type":"ContainerStarted","Data":"5daae85166930a9aae9b44b785e6958ca6749d3451b4bdf0b1e737f392432c62"} Jul 10 22:28:53.429868 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.429839 2565 generic.go:356] "Generic (PLEG): container finished" podID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" containerID="71e5c63f880a2530ddc560888635b51490dcbe9c690e876b23683bd2f4034f56" exitCode=0 Jul 10 22:28:53.430018 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.429886 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:53.430018 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.429886 2565 scope.go:117] "RemoveContainer" containerID="71e5c63f880a2530ddc560888635b51490dcbe9c690e876b23683bd2f4034f56" Jul 10 22:28:53.430909 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:53.430875 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6wl84\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:53.431387 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.431363 2565 generic.go:356] "Generic (PLEG): container finished" podID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" containerID="9afc9654afbde2e90e0ac3f9ba2b8b65f2c13170682730bed6603622707656b7" exitCode=0 Jul 10 22:28:53.431510 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.431403 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:53.433365 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.433325 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzp7b" event={"ID":"2ffca01f-3f3a-496c-93ac-0d46d9f27915","Type":"ContainerStarted","Data":"1415eb257a807444ff699a00f5e359a284879299d14c2385626b1e7750ed1dc8"} Jul 10 22:28:53.433472 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.433377 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzp7b" event={"ID":"2ffca01f-3f3a-496c-93ac-0d46d9f27915","Type":"ContainerStarted","Data":"2f57bbc8cb40717318b7621b2640213eb303fb9ff99af1b9ab783f73fe354c39"} Jul 10 22:28:53.433472 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.433417 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzp7b" Jul 10 22:28:53.433596 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:53.433530 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:53.439932 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.439917 2565 scope.go:117] "RemoveContainer" containerID="71e5c63f880a2530ddc560888635b51490dcbe9c690e876b23683bd2f4034f56" Jul 10 22:28:53.440184 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:53.440167 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e5c63f880a2530ddc560888635b51490dcbe9c690e876b23683bd2f4034f56\": container with ID starting with 71e5c63f880a2530ddc560888635b51490dcbe9c690e876b23683bd2f4034f56 not found: ID does not exist" containerID="71e5c63f880a2530ddc560888635b51490dcbe9c690e876b23683bd2f4034f56" Jul 10 22:28:53.440257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.440198 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e5c63f880a2530ddc560888635b51490dcbe9c690e876b23683bd2f4034f56"} err="failed to get container status \"71e5c63f880a2530ddc560888635b51490dcbe9c690e876b23683bd2f4034f56\": rpc error: code = NotFound desc = could not find container \"71e5c63f880a2530ddc560888635b51490dcbe9c690e876b23683bd2f4034f56\": container with ID starting with 71e5c63f880a2530ddc560888635b51490dcbe9c690e876b23683bd2f4034f56 not found: ID does not exist" Jul 10 22:28:53.440257 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.440222 2565 scope.go:117] "RemoveContainer" containerID="9afc9654afbde2e90e0ac3f9ba2b8b65f2c13170682730bed6603622707656b7" Jul 10 22:28:53.448189 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.448166 2565 scope.go:117] "RemoveContainer" containerID="9afc9654afbde2e90e0ac3f9ba2b8b65f2c13170682730bed6603622707656b7" Jul 10 22:28:53.448509 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:53.448491 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9afc9654afbde2e90e0ac3f9ba2b8b65f2c13170682730bed6603622707656b7\": container with ID starting with 9afc9654afbde2e90e0ac3f9ba2b8b65f2c13170682730bed6603622707656b7 not found: ID does not exist" containerID="9afc9654afbde2e90e0ac3f9ba2b8b65f2c13170682730bed6603622707656b7" Jul 10 22:28:53.448569 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.448521 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afc9654afbde2e90e0ac3f9ba2b8b65f2c13170682730bed6603622707656b7"} err="failed to get container status \"9afc9654afbde2e90e0ac3f9ba2b8b65f2c13170682730bed6603622707656b7\": rpc error: code = NotFound desc = could not find container \"9afc9654afbde2e90e0ac3f9ba2b8b65f2c13170682730bed6603622707656b7\": container with ID starting with 9afc9654afbde2e90e0ac3f9ba2b8b65f2c13170682730bed6603622707656b7 not found: ID does not exist" Jul 10 22:28:53.459253 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.459206 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zxzzm" podStartSLOduration=1.605940114 podStartE2EDuration="24.4591923s" totalImagesPullingTime="22.853252186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:28:53.45778755 +0000 UTC m=+397.290671932" watchObservedRunningTime="2026-07-10 22:28:53.4591923 +0000 UTC m=+397.292076682" Jul 10 22:28:53.463084 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:53.463061 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6wl84\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6wl84" Jul 10 22:28:53.466683 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:28:53.466660 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht2r6\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht2r6" Jul 10 22:28:53.488447 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.488407 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kps2t\" (UniqueName: \"kubernetes.io/projected/ea5f39d9-0d24-4448-9b13-b8d8e3bd1660-kube-api-access-kps2t\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:53.488447 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.488440 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea5f39d9-0d24-4448-9b13-b8d8e3bd1660-extensions-socket-volume\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:53.488611 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.488457 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf8kw\" (UniqueName: \"kubernetes.io/projected/60d1a470-ca0b-4320-b949-7ec0613cbd0d-kube-api-access-lf8kw\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:28:53.512305 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.512251 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzp7b" podStartSLOduration=13.512233418 podStartE2EDuration="13.512233418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:28:53.510623107 +0000 UTC m=+397.343507491" watchObservedRunningTime="2026-07-10 22:28:53.512233418 +0000 UTC m=+397.345117800" Jul 10 22:28:53.691005 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.690928 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60d1a470-ca0b-4320-b949-7ec0613cbd0d" path="/var/lib/kubelet/pods/60d1a470-ca0b-4320-b949-7ec0613cbd0d/volumes" Jul 10 22:28:53.691268 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:53.691254 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea5f39d9-0d24-4448-9b13-b8d8e3bd1660" path="/var/lib/kubelet/pods/ea5f39d9-0d24-4448-9b13-b8d8e3bd1660/volumes" Jul 10 22:28:56.049247 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:56.049212 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:56.107617 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:56.107586 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:57.315315 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.315282 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-757cbd9596-qs97t"] Jul 10 22:28:57.320160 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.320138 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.329009 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.328985 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-757cbd9596-qs97t"] Jul 10 22:28:57.376704 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.376667 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:28:57.420981 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.420950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-oauth-serving-cert\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.421123 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.420999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-trusted-ca-bundle\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.421123 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.421052 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-console-serving-cert\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.421123 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.421082 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9cx\" (UniqueName: \"kubernetes.io/projected/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-kube-api-access-xq9cx\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.421279 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.421139 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-console-oauth-config\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.421279 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.421188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-console-config\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.421279 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.421213 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-service-ca\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.522071 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.522038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9cx\" (UniqueName: \"kubernetes.io/projected/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-kube-api-access-xq9cx\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.522248 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.522093 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-console-oauth-config\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.522248 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.522209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-console-config\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.522359 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.522248 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-service-ca\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.522425 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.522403 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-oauth-serving-cert\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.522495 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.522481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-trusted-ca-bundle\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.522554 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.522530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-console-serving-cert\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.523162 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.523048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-service-ca\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.523162 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.523098 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-console-config\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.523162 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.523112 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-oauth-serving-cert\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.523372 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.523302 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-trusted-ca-bundle\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.524749 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.524722 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-console-oauth-config\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.524749 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.524746 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-console-serving-cert\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.533776 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.533749 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9cx\" (UniqueName: \"kubernetes.io/projected/97cbba24-8d97-4bd2-9a3d-58e79ba08b92-kube-api-access-xq9cx\") pod \"console-757cbd9596-qs97t\" (UID: \"97cbba24-8d97-4bd2-9a3d-58e79ba08b92\") " pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.629499 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.629409 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:28:57.777349 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:57.777321 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-757cbd9596-qs97t"] Jul 10 22:28:57.779244 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:28:57.779217 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cbba24_8d97_4bd2_9a3d_58e79ba08b92.slice/crio-b8664bbbba3f652607a02a8fc688e6c3facecb095bcc5538da8378e937672784 WatchSource:0}: Error finding container b8664bbbba3f652607a02a8fc688e6c3facecb095bcc5538da8378e937672784: Status 404 returned error can't find the container with id b8664bbbba3f652607a02a8fc688e6c3facecb095bcc5538da8378e937672784 Jul 10 22:28:58.461734 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:58.461696 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-757cbd9596-qs97t" event={"ID":"97cbba24-8d97-4bd2-9a3d-58e79ba08b92","Type":"ContainerStarted","Data":"7906b758263c4a445f6afb67111159dc3d52fe19e31fc222dc49992525f63065"} Jul 10 22:28:58.462133 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:58.461750 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-757cbd9596-qs97t" event={"ID":"97cbba24-8d97-4bd2-9a3d-58e79ba08b92","Type":"ContainerStarted","Data":"b8664bbbba3f652607a02a8fc688e6c3facecb095bcc5538da8378e937672784"} Jul 10 22:28:58.480720 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:28:58.480667 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-757cbd9596-qs97t" podStartSLOduration=1.480651399 podStartE2EDuration="1.480651399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:28:58.478741029 +0000 UTC m=+402.311625412" watchObservedRunningTime="2026-07-10 22:28:58.480651399 +0000 UTC m=+402.313535781" Jul 10 22:29:02.107016 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:02.106960 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:02.170958 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:02.170927 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:04.443230 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:04.443199 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzp7b" Jul 10 22:29:06.042552 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:06.042512 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:07.630400 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:07.630356 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:29:07.630781 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:07.630415 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:29:07.635318 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:07.635294 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:29:08.517035 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:08.517002 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-757cbd9596-qs97t" Jul 10 22:29:08.567643 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:08.567611 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dd59c8566-xr5nr"] Jul 10 22:29:08.666776 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:08.666740 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:08.761728 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:08.761695 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:08.818496 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:08.818415 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:11.659515 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:11.659474 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:11.762183 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:11.762151 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:12.679271 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.679238 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx"] Jul 10 22:29:12.685003 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.684980 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.687318 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.687299 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dlht7\"" Jul 10 22:29:12.697492 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.697466 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx"] Jul 10 22:29:12.698689 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.698664 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:12.755121 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.755077 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.755121 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.755121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.755328 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.755214 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a688576f-a240-451c-84a0-31ef6c45f6a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.755328 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.755250 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.755328 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.755278 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a688576f-a240-451c-84a0-31ef6c45f6a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.755328 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.755298 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.755328 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.755314 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcpct\" (UniqueName: \"kubernetes.io/projected/a688576f-a240-451c-84a0-31ef6c45f6a9-kube-api-access-gcpct\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.755520 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.755359 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.755520 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.755421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a688576f-a240-451c-84a0-31ef6c45f6a9-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.839644 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.839606 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:12.856779 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.856751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a688576f-a240-451c-84a0-31ef6c45f6a9-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.856891 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.856817 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.856891 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.856843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.856973 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.856892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a688576f-a240-451c-84a0-31ef6c45f6a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.856973 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.856956 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.857122 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.856988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a688576f-a240-451c-84a0-31ef6c45f6a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.857122 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.857019 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.857122 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.857042 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcpct\" (UniqueName: \"kubernetes.io/projected/a688576f-a240-451c-84a0-31ef6c45f6a9-kube-api-access-gcpct\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.857122 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.857108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.857327 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.857268 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.857396 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.857329 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.857396 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.857372 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.857560 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.857540 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.857827 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.857788 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a688576f-a240-451c-84a0-31ef6c45f6a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.859253 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.859231 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a688576f-a240-451c-84a0-31ef6c45f6a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.859356 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.859338 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a688576f-a240-451c-84a0-31ef6c45f6a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.864284 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.864264 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a688576f-a240-451c-84a0-31ef6c45f6a9-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.864866 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.864848 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcpct\" (UniqueName: \"kubernetes.io/projected/a688576f-a240-451c-84a0-31ef6c45f6a9-kube-api-access-gcpct\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sgnbx\" (UID: \"a688576f-a240-451c-84a0-31ef6c45f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:12.997454 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:12.997414 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:13.160664 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:13.160633 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx"] Jul 10 22:29:13.161858 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:29:13.161829 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda688576f_a240_451c_84a0_31ef6c45f6a9.slice/crio-40070b9502d79a7fb9388e086ef6d12276df4b0421493683bf220e14d1e2699b WatchSource:0}: Error finding container 40070b9502d79a7fb9388e086ef6d12276df4b0421493683bf220e14d1e2699b: Status 404 returned error can't find the container with id 40070b9502d79a7fb9388e086ef6d12276df4b0421493683bf220e14d1e2699b Jul 10 22:29:13.164020 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:13.163985 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:29:13.164081 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:13.164056 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:29:13.164124 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:13.164085 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:29:13.536867 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:13.536767 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" event={"ID":"a688576f-a240-451c-84a0-31ef6c45f6a9","Type":"ContainerStarted","Data":"9d66b2c1b32a2e49ed1e7a0c293518d7c3317bce5a4c474d27b054de870df62f"} Jul 10 22:29:13.536867 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:13.536850 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" event={"ID":"a688576f-a240-451c-84a0-31ef6c45f6a9","Type":"ContainerStarted","Data":"40070b9502d79a7fb9388e086ef6d12276df4b0421493683bf220e14d1e2699b"} Jul 10 22:29:13.556486 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:13.556430 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" podStartSLOduration=1.5564128739999998 podStartE2EDuration="1.556412874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:29:13.553954021 +0000 UTC m=+417.386838407" watchObservedRunningTime="2026-07-10 22:29:13.556412874 +0000 UTC m=+417.389297257" Jul 10 22:29:13.997771 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:13.997738 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:14.618221 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.618188 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:14.622852 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.622826 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xftv2"] Jul 10 22:29:14.643719 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.643686 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xftv2"] Jul 10 22:29:14.643866 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.643824 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" Jul 10 22:29:14.646986 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.646967 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 22:29:14.715901 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.715869 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xftv2"] Jul 10 22:29:14.721655 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.721630 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:14.773289 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.773255 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/354bf0ee-1a22-4980-a422-8a435f0bc05f-config-file\") pod \"limitador-limitador-64c94fd64f-xftv2\" (UID: \"354bf0ee-1a22-4980-a422-8a435f0bc05f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" Jul 10 22:29:14.773439 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.773311 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ppsr\" (UniqueName: \"kubernetes.io/projected/354bf0ee-1a22-4980-a422-8a435f0bc05f-kube-api-access-7ppsr\") pod \"limitador-limitador-64c94fd64f-xftv2\" (UID: \"354bf0ee-1a22-4980-a422-8a435f0bc05f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" Jul 10 22:29:14.780712 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.780687 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:14.874395 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.874314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/354bf0ee-1a22-4980-a422-8a435f0bc05f-config-file\") pod \"limitador-limitador-64c94fd64f-xftv2\" (UID: \"354bf0ee-1a22-4980-a422-8a435f0bc05f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" Jul 10 22:29:14.874395 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.874369 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ppsr\" (UniqueName: \"kubernetes.io/projected/354bf0ee-1a22-4980-a422-8a435f0bc05f-kube-api-access-7ppsr\") pod \"limitador-limitador-64c94fd64f-xftv2\" (UID: \"354bf0ee-1a22-4980-a422-8a435f0bc05f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" Jul 10 22:29:14.875176 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.875155 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/354bf0ee-1a22-4980-a422-8a435f0bc05f-config-file\") pod \"limitador-limitador-64c94fd64f-xftv2\" (UID: \"354bf0ee-1a22-4980-a422-8a435f0bc05f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" Jul 10 22:29:14.881885 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.881861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ppsr\" (UniqueName: \"kubernetes.io/projected/354bf0ee-1a22-4980-a422-8a435f0bc05f-kube-api-access-7ppsr\") pod \"limitador-limitador-64c94fd64f-xftv2\" (UID: \"354bf0ee-1a22-4980-a422-8a435f0bc05f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" Jul 10 22:29:14.954892 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:14.954858 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" Jul 10 22:29:15.004105 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:15.004071 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:15.093484 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:15.093461 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xftv2"] Jul 10 22:29:15.094717 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:29:15.094692 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354bf0ee_1a22_4980_a422_8a435f0bc05f.slice/crio-215b153ff9e353d7886a84ff01ef581ae129cab546d30d25b98c181e491f3d4f WatchSource:0}: Error finding container 215b153ff9e353d7886a84ff01ef581ae129cab546d30d25b98c181e491f3d4f: Status 404 returned error can't find the container with id 215b153ff9e353d7886a84ff01ef581ae129cab546d30d25b98c181e491f3d4f Jul 10 22:29:15.449767 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:15.449735 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:15.548415 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:15.548378 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" event={"ID":"354bf0ee-1a22-4980-a422-8a435f0bc05f","Type":"ContainerStarted","Data":"215b153ff9e353d7886a84ff01ef581ae129cab546d30d25b98c181e491f3d4f"} Jul 10 22:29:15.548699 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:15.548676 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:15.549658 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:15.549637 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:29:15.549925 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:15.549906 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:15.685563 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:15.685529 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:15.748862 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:15.748832 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:19.570347 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:19.570312 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" event={"ID":"354bf0ee-1a22-4980-a422-8a435f0bc05f","Type":"ContainerStarted","Data":"0640bf817c6e098d415b39e97b622c96ddd261126d570f9475c0cda0443cc888"} Jul 10 22:29:19.570754 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:19.570460 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" Jul 10 22:29:19.587090 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:19.587041 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" podStartSLOduration=1.7309134739999998 podStartE2EDuration="5.587024872s" totalImagesPullingTime="3.856111398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:29:19.585632891 +0000 UTC m=+423.418517300" watchObservedRunningTime="2026-07-10 22:29:19.587024872 +0000 UTC m=+423.419909556" Jul 10 22:29:23.669054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:23.669012 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:23.727554 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:23.727476 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:29.858261 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:29.858227 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:29.960131 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:29.960094 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:30.060142 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:30.060104 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:30.119488 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:30.119417 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:30.575519 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:30.575492 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" Jul 10 22:29:31.844046 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:31.844016 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xftv2"] Jul 10 22:29:31.844522 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:31.844237 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" podUID="354bf0ee-1a22-4980-a422-8a435f0bc05f" containerName="limitador" containerID="cri-o://0640bf817c6e098d415b39e97b622c96ddd261126d570f9475c0cda0443cc888" gracePeriod=30 Jul 10 22:29:32.075011 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.074961 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:32.133602 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.133521 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:32.385127 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.385059 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" Jul 10 22:29:32.429246 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.429216 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ppsr\" (UniqueName: \"kubernetes.io/projected/354bf0ee-1a22-4980-a422-8a435f0bc05f-kube-api-access-7ppsr\") pod \"354bf0ee-1a22-4980-a422-8a435f0bc05f\" (UID: \"354bf0ee-1a22-4980-a422-8a435f0bc05f\") " Jul 10 22:29:32.429390 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.429268 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/354bf0ee-1a22-4980-a422-8a435f0bc05f-config-file\") pod \"354bf0ee-1a22-4980-a422-8a435f0bc05f\" (UID: \"354bf0ee-1a22-4980-a422-8a435f0bc05f\") " Jul 10 22:29:32.429658 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.429634 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354bf0ee-1a22-4980-a422-8a435f0bc05f-config-file" (OuterVolumeSpecName: "config-file") pod "354bf0ee-1a22-4980-a422-8a435f0bc05f" (UID: "354bf0ee-1a22-4980-a422-8a435f0bc05f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:29:32.431283 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.431266 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354bf0ee-1a22-4980-a422-8a435f0bc05f-kube-api-access-7ppsr" (OuterVolumeSpecName: "kube-api-access-7ppsr") pod "354bf0ee-1a22-4980-a422-8a435f0bc05f" (UID: "354bf0ee-1a22-4980-a422-8a435f0bc05f"). InnerVolumeSpecName "kube-api-access-7ppsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:29:32.530667 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.530642 2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/354bf0ee-1a22-4980-a422-8a435f0bc05f-config-file\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:29:32.530667 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.530666 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ppsr\" (UniqueName: \"kubernetes.io/projected/354bf0ee-1a22-4980-a422-8a435f0bc05f-kube-api-access-7ppsr\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:29:32.635495 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.635417 2565 generic.go:356] "Generic (PLEG): container finished" podID="354bf0ee-1a22-4980-a422-8a435f0bc05f" containerID="0640bf817c6e098d415b39e97b622c96ddd261126d570f9475c0cda0443cc888" exitCode=0 Jul 10 22:29:32.635495 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.635458 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" Jul 10 22:29:32.635657 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.635495 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" event={"ID":"354bf0ee-1a22-4980-a422-8a435f0bc05f","Type":"ContainerDied","Data":"0640bf817c6e098d415b39e97b622c96ddd261126d570f9475c0cda0443cc888"} Jul 10 22:29:32.635657 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.635545 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-xftv2" event={"ID":"354bf0ee-1a22-4980-a422-8a435f0bc05f","Type":"ContainerDied","Data":"215b153ff9e353d7886a84ff01ef581ae129cab546d30d25b98c181e491f3d4f"} Jul 10 22:29:32.635657 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.635566 2565 scope.go:117] "RemoveContainer" containerID="0640bf817c6e098d415b39e97b622c96ddd261126d570f9475c0cda0443cc888" Jul 10 22:29:32.644234 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.644218 2565 scope.go:117] "RemoveContainer" containerID="0640bf817c6e098d415b39e97b622c96ddd261126d570f9475c0cda0443cc888" Jul 10 22:29:32.644469 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:29:32.644450 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0640bf817c6e098d415b39e97b622c96ddd261126d570f9475c0cda0443cc888\": container with ID starting with 0640bf817c6e098d415b39e97b622c96ddd261126d570f9475c0cda0443cc888 not found: ID does not exist" containerID="0640bf817c6e098d415b39e97b622c96ddd261126d570f9475c0cda0443cc888" Jul 10 22:29:32.644523 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.644477 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0640bf817c6e098d415b39e97b622c96ddd261126d570f9475c0cda0443cc888"} err="failed to get container status \"0640bf817c6e098d415b39e97b622c96ddd261126d570f9475c0cda0443cc888\": rpc error: code = NotFound desc = could not find container \"0640bf817c6e098d415b39e97b622c96ddd261126d570f9475c0cda0443cc888\": container with ID starting with 0640bf817c6e098d415b39e97b622c96ddd261126d570f9475c0cda0443cc888 not found: ID does not exist" Jul 10 22:29:32.656093 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.656074 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xftv2"] Jul 10 22:29:32.659316 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:32.659296 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xftv2"] Jul 10 22:29:33.589619 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.589583 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5dd59c8566-xr5nr" podUID="535acecf-ae50-4ad5-81f9-ead7d79df068" containerName="console" containerID="cri-o://ac1d36fd2f05bdb74c79c72b2d351325fed05a6336f48dc7ce400d58f7988d44" gracePeriod=15 Jul 10 22:29:33.690572 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.690535 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="354bf0ee-1a22-4980-a422-8a435f0bc05f" path="/var/lib/kubelet/pods/354bf0ee-1a22-4980-a422-8a435f0bc05f/volumes" Jul 10 22:29:33.842845 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.842769 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:29:33.943045 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.943002 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x44bb\" (UniqueName: \"kubernetes.io/projected/535acecf-ae50-4ad5-81f9-ead7d79df068-kube-api-access-x44bb\") pod \"535acecf-ae50-4ad5-81f9-ead7d79df068\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " Jul 10 22:29:33.943236 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.943073 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-console-config\") pod \"535acecf-ae50-4ad5-81f9-ead7d79df068\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " Jul 10 22:29:33.943236 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.943093 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/535acecf-ae50-4ad5-81f9-ead7d79df068-console-oauth-config\") pod \"535acecf-ae50-4ad5-81f9-ead7d79df068\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " Jul 10 22:29:33.943236 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.943117 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/535acecf-ae50-4ad5-81f9-ead7d79df068-console-serving-cert\") pod \"535acecf-ae50-4ad5-81f9-ead7d79df068\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " Jul 10 22:29:33.943236 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.943144 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-oauth-serving-cert\") pod \"535acecf-ae50-4ad5-81f9-ead7d79df068\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " Jul 10 22:29:33.943236 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.943166 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-service-ca\") pod \"535acecf-ae50-4ad5-81f9-ead7d79df068\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " Jul 10 22:29:33.943236 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.943191 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-trusted-ca-bundle\") pod \"535acecf-ae50-4ad5-81f9-ead7d79df068\" (UID: \"535acecf-ae50-4ad5-81f9-ead7d79df068\") " Jul 10 22:29:33.943684 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.943656 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "535acecf-ae50-4ad5-81f9-ead7d79df068" (UID: "535acecf-ae50-4ad5-81f9-ead7d79df068"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:29:33.943684 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.943666 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-service-ca" (OuterVolumeSpecName: "service-ca") pod "535acecf-ae50-4ad5-81f9-ead7d79df068" (UID: "535acecf-ae50-4ad5-81f9-ead7d79df068"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:29:33.943841 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.943682 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "535acecf-ae50-4ad5-81f9-ead7d79df068" (UID: "535acecf-ae50-4ad5-81f9-ead7d79df068"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:29:33.943841 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.943705 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-console-config" (OuterVolumeSpecName: "console-config") pod "535acecf-ae50-4ad5-81f9-ead7d79df068" (UID: "535acecf-ae50-4ad5-81f9-ead7d79df068"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:29:33.945344 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.945318 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535acecf-ae50-4ad5-81f9-ead7d79df068-kube-api-access-x44bb" (OuterVolumeSpecName: "kube-api-access-x44bb") pod "535acecf-ae50-4ad5-81f9-ead7d79df068" (UID: "535acecf-ae50-4ad5-81f9-ead7d79df068"). InnerVolumeSpecName "kube-api-access-x44bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:29:33.945344 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.945331 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/535acecf-ae50-4ad5-81f9-ead7d79df068-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "535acecf-ae50-4ad5-81f9-ead7d79df068" (UID: "535acecf-ae50-4ad5-81f9-ead7d79df068"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:29:33.945468 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:33.945372 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/535acecf-ae50-4ad5-81f9-ead7d79df068-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "535acecf-ae50-4ad5-81f9-ead7d79df068" (UID: "535acecf-ae50-4ad5-81f9-ead7d79df068"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:29:34.044372 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.044329 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-service-ca\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:29:34.044372 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.044364 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-trusted-ca-bundle\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:29:34.044372 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.044376 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x44bb\" (UniqueName: \"kubernetes.io/projected/535acecf-ae50-4ad5-81f9-ead7d79df068-kube-api-access-x44bb\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:29:34.044620 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.044388 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-console-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:29:34.044620 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.044400 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/535acecf-ae50-4ad5-81f9-ead7d79df068-console-oauth-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:29:34.044620 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.044411 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/535acecf-ae50-4ad5-81f9-ead7d79df068-console-serving-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:29:34.044620 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.044422 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/535acecf-ae50-4ad5-81f9-ead7d79df068-oauth-serving-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:29:34.646330 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.646276 2565 generic.go:356] "Generic (PLEG): container finished" podID="535acecf-ae50-4ad5-81f9-ead7d79df068" containerID="ac1d36fd2f05bdb74c79c72b2d351325fed05a6336f48dc7ce400d58f7988d44" exitCode=0 Jul 10 22:29:34.646330 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.646329 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dd59c8566-xr5nr" Jul 10 22:29:34.646815 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.646361 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dd59c8566-xr5nr" event={"ID":"535acecf-ae50-4ad5-81f9-ead7d79df068","Type":"ContainerDied","Data":"ac1d36fd2f05bdb74c79c72b2d351325fed05a6336f48dc7ce400d58f7988d44"} Jul 10 22:29:34.646815 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.646423 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dd59c8566-xr5nr" event={"ID":"535acecf-ae50-4ad5-81f9-ead7d79df068","Type":"ContainerDied","Data":"6f2b7bee71fa3d3c4fad25d3c41829300194bf459526af2d6668937e484e9cf8"} Jul 10 22:29:34.646815 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.646454 2565 scope.go:117] "RemoveContainer" containerID="ac1d36fd2f05bdb74c79c72b2d351325fed05a6336f48dc7ce400d58f7988d44" Jul 10 22:29:34.655490 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.655474 2565 scope.go:117] "RemoveContainer" containerID="ac1d36fd2f05bdb74c79c72b2d351325fed05a6336f48dc7ce400d58f7988d44" Jul 10 22:29:34.655709 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:29:34.655687 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1d36fd2f05bdb74c79c72b2d351325fed05a6336f48dc7ce400d58f7988d44\": container with ID starting with ac1d36fd2f05bdb74c79c72b2d351325fed05a6336f48dc7ce400d58f7988d44 not found: ID does not exist" containerID="ac1d36fd2f05bdb74c79c72b2d351325fed05a6336f48dc7ce400d58f7988d44" Jul 10 22:29:34.655776 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.655723 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1d36fd2f05bdb74c79c72b2d351325fed05a6336f48dc7ce400d58f7988d44"} err="failed to get container status \"ac1d36fd2f05bdb74c79c72b2d351325fed05a6336f48dc7ce400d58f7988d44\": rpc error: code = NotFound desc = could not find container \"ac1d36fd2f05bdb74c79c72b2d351325fed05a6336f48dc7ce400d58f7988d44\": container with ID starting with ac1d36fd2f05bdb74c79c72b2d351325fed05a6336f48dc7ce400d58f7988d44 not found: ID does not exist" Jul 10 22:29:34.666509 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.666485 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dd59c8566-xr5nr"] Jul 10 22:29:34.672022 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:34.671997 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dd59c8566-xr5nr"] Jul 10 22:29:35.693001 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:35.691426 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="535acecf-ae50-4ad5-81f9-ead7d79df068" path="/var/lib/kubelet/pods/535acecf-ae50-4ad5-81f9-ead7d79df068/volumes" Jul 10 22:29:35.788294 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:35.788262 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:35.888150 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:35.888118 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:36.238069 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.238037 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs"] Jul 10 22:29:36.238366 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.238355 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="535acecf-ae50-4ad5-81f9-ead7d79df068" containerName="console" Jul 10 22:29:36.238408 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.238369 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="535acecf-ae50-4ad5-81f9-ead7d79df068" containerName="console" Jul 10 22:29:36.238408 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.238387 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354bf0ee-1a22-4980-a422-8a435f0bc05f" containerName="limitador" Jul 10 22:29:36.238408 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.238392 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="354bf0ee-1a22-4980-a422-8a435f0bc05f" containerName="limitador" Jul 10 22:29:36.238499 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.238449 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="535acecf-ae50-4ad5-81f9-ead7d79df068" containerName="console" Jul 10 22:29:36.238499 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.238458 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="354bf0ee-1a22-4980-a422-8a435f0bc05f" containerName="limitador" Jul 10 22:29:36.242996 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.242971 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs" Jul 10 22:29:36.245329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.245289 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 22:29:36.245452 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.245382 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bjrvc\"" Jul 10 22:29:36.245452 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.245407 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 22:29:36.246337 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.246316 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 22:29:36.249023 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.249002 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs"] Jul 10 22:29:36.263369 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.263345 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/124b9f94-653b-441f-a4f9-2d8153c39cab-data\") pod \"postgres-69fb4594cf-bzpbs\" (UID: \"124b9f94-653b-441f-a4f9-2d8153c39cab\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs" Jul 10 22:29:36.263501 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.263406 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stgw9\" (UniqueName: \"kubernetes.io/projected/124b9f94-653b-441f-a4f9-2d8153c39cab-kube-api-access-stgw9\") pod \"postgres-69fb4594cf-bzpbs\" (UID: \"124b9f94-653b-441f-a4f9-2d8153c39cab\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs" Jul 10 22:29:36.273247 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.273223 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:36.364212 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.364177 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stgw9\" (UniqueName: \"kubernetes.io/projected/124b9f94-653b-441f-a4f9-2d8153c39cab-kube-api-access-stgw9\") pod \"postgres-69fb4594cf-bzpbs\" (UID: \"124b9f94-653b-441f-a4f9-2d8153c39cab\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs" Jul 10 22:29:36.364390 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.364256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/124b9f94-653b-441f-a4f9-2d8153c39cab-data\") pod \"postgres-69fb4594cf-bzpbs\" (UID: \"124b9f94-653b-441f-a4f9-2d8153c39cab\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs" Jul 10 22:29:36.364597 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.364579 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/124b9f94-653b-441f-a4f9-2d8153c39cab-data\") pod \"postgres-69fb4594cf-bzpbs\" (UID: \"124b9f94-653b-441f-a4f9-2d8153c39cab\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs" Jul 10 22:29:36.372452 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.372421 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stgw9\" (UniqueName: \"kubernetes.io/projected/124b9f94-653b-441f-a4f9-2d8153c39cab-kube-api-access-stgw9\") pod \"postgres-69fb4594cf-bzpbs\" (UID: \"124b9f94-653b-441f-a4f9-2d8153c39cab\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs" Jul 10 22:29:36.556536 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.556447 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs" Jul 10 22:29:36.685833 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:36.685630 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs"] Jul 10 22:29:36.688463 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:29:36.688431 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124b9f94_653b_441f_a4f9_2d8153c39cab.slice/crio-95e9cc607017248079a4758896f559a31bcb7cc2174adcbe473b660eff8c4d4c WatchSource:0}: Error finding container 95e9cc607017248079a4758896f559a31bcb7cc2174adcbe473b660eff8c4d4c: Status 404 returned error can't find the container with id 95e9cc607017248079a4758896f559a31bcb7cc2174adcbe473b660eff8c4d4c Jul 10 22:29:37.664803 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:37.664758 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs" event={"ID":"124b9f94-653b-441f-a4f9-2d8153c39cab","Type":"ContainerStarted","Data":"95e9cc607017248079a4758896f559a31bcb7cc2174adcbe473b660eff8c4d4c"} Jul 10 22:29:42.698350 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:42.698315 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs" event={"ID":"124b9f94-653b-441f-a4f9-2d8153c39cab","Type":"ContainerStarted","Data":"41cd89620fc288d81d073a52cd0ae0a3b0f1cb068f16c3fcb3bbf0f8bd9ecfa8"} Jul 10 22:29:42.698730 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:42.698437 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs" Jul 10 22:29:42.714141 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:42.714094 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs" podStartSLOduration=1.389225443 podStartE2EDuration="6.714079635s" totalImagesPullingTime="5.324854192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:29:42.713676048 +0000 UTC m=+446.546560431" watchObservedRunningTime="2026-07-10 22:29:42.714079635 +0000 UTC m=+446.546964017" Jul 10 22:29:48.732123 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:48.732085 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bzpbs" Jul 10 22:29:49.039158 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:49.039083 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:49.141708 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:49.141670 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:55.258479 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.258428 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:55.273101 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.273071 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f586fdfd-crh7d"] Jul 10 22:29:55.293221 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.293187 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f586fdfd-crh7d"] Jul 10 22:29:55.293351 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.293318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f586fdfd-crh7d" Jul 10 22:29:55.295832 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.295812 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-n68dc\"" Jul 10 22:29:55.296075 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.296061 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 22:29:55.296301 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.296287 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 22:29:55.331751 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.331724 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:55.382568 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.382539 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:55.439788 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.439760 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4tqh\" (UniqueName: \"kubernetes.io/projected/0fae49b8-a6f3-41ab-bad5-753a8c39df6b-kube-api-access-h4tqh\") pod \"maas-controller-7f586fdfd-crh7d\" (UID: \"0fae49b8-a6f3-41ab-bad5-753a8c39df6b\") " pod="opendatahub/maas-controller-7f586fdfd-crh7d" Jul 10 22:29:55.439966 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.439826 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fae49b8-a6f3-41ab-bad5-753a8c39df6b-webhook-cert\") pod \"maas-controller-7f586fdfd-crh7d\" (UID: \"0fae49b8-a6f3-41ab-bad5-753a8c39df6b\") " pod="opendatahub/maas-controller-7f586fdfd-crh7d" Jul 10 22:29:55.440723 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.440705 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-795cbfd498-4rrtw"] Jul 10 22:29:55.443225 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.443204 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-795cbfd498-4rrtw" Jul 10 22:29:55.452436 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.452406 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-795cbfd498-4rrtw"] Jul 10 22:29:55.500969 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.500938 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:55.541329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.541248 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdnl\" (UniqueName: \"kubernetes.io/projected/a3a6e213-1eed-47bc-a5b3-8da897663feb-kube-api-access-bkdnl\") pod \"maas-controller-795cbfd498-4rrtw\" (UID: \"a3a6e213-1eed-47bc-a5b3-8da897663feb\") " pod="opendatahub/maas-controller-795cbfd498-4rrtw" Jul 10 22:29:55.541329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.541286 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4tqh\" (UniqueName: \"kubernetes.io/projected/0fae49b8-a6f3-41ab-bad5-753a8c39df6b-kube-api-access-h4tqh\") pod \"maas-controller-7f586fdfd-crh7d\" (UID: \"0fae49b8-a6f3-41ab-bad5-753a8c39df6b\") " pod="opendatahub/maas-controller-7f586fdfd-crh7d" Jul 10 22:29:55.541503 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.541359 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3a6e213-1eed-47bc-a5b3-8da897663feb-webhook-cert\") pod \"maas-controller-795cbfd498-4rrtw\" (UID: \"a3a6e213-1eed-47bc-a5b3-8da897663feb\") " pod="opendatahub/maas-controller-795cbfd498-4rrtw" Jul 10 22:29:55.541503 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.541400 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fae49b8-a6f3-41ab-bad5-753a8c39df6b-webhook-cert\") pod \"maas-controller-7f586fdfd-crh7d\" (UID: \"0fae49b8-a6f3-41ab-bad5-753a8c39df6b\") " pod="opendatahub/maas-controller-7f586fdfd-crh7d" Jul 10 22:29:55.543787 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.543771 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fae49b8-a6f3-41ab-bad5-753a8c39df6b-webhook-cert\") pod \"maas-controller-7f586fdfd-crh7d\" (UID: \"0fae49b8-a6f3-41ab-bad5-753a8c39df6b\") " pod="opendatahub/maas-controller-7f586fdfd-crh7d" Jul 10 22:29:55.549395 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.549373 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4tqh\" (UniqueName: \"kubernetes.io/projected/0fae49b8-a6f3-41ab-bad5-753a8c39df6b-kube-api-access-h4tqh\") pod \"maas-controller-7f586fdfd-crh7d\" (UID: \"0fae49b8-a6f3-41ab-bad5-753a8c39df6b\") " pod="opendatahub/maas-controller-7f586fdfd-crh7d" Jul 10 22:29:55.606690 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.606652 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f586fdfd-crh7d" Jul 10 22:29:55.641895 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.641856 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdnl\" (UniqueName: \"kubernetes.io/projected/a3a6e213-1eed-47bc-a5b3-8da897663feb-kube-api-access-bkdnl\") pod \"maas-controller-795cbfd498-4rrtw\" (UID: \"a3a6e213-1eed-47bc-a5b3-8da897663feb\") " pod="opendatahub/maas-controller-795cbfd498-4rrtw" Jul 10 22:29:55.642072 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.641922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3a6e213-1eed-47bc-a5b3-8da897663feb-webhook-cert\") pod \"maas-controller-795cbfd498-4rrtw\" (UID: \"a3a6e213-1eed-47bc-a5b3-8da897663feb\") " pod="opendatahub/maas-controller-795cbfd498-4rrtw" Jul 10 22:29:55.644514 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.644451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3a6e213-1eed-47bc-a5b3-8da897663feb-webhook-cert\") pod \"maas-controller-795cbfd498-4rrtw\" (UID: \"a3a6e213-1eed-47bc-a5b3-8da897663feb\") " pod="opendatahub/maas-controller-795cbfd498-4rrtw" Jul 10 22:29:55.649963 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.649935 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdnl\" (UniqueName: \"kubernetes.io/projected/a3a6e213-1eed-47bc-a5b3-8da897663feb-kube-api-access-bkdnl\") pod \"maas-controller-795cbfd498-4rrtw\" (UID: \"a3a6e213-1eed-47bc-a5b3-8da897663feb\") " pod="opendatahub/maas-controller-795cbfd498-4rrtw" Jul 10 22:29:55.736457 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.736434 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f586fdfd-crh7d"] Jul 10 22:29:55.738327 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:29:55.738301 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fae49b8_a6f3_41ab_bad5_753a8c39df6b.slice/crio-8cc6b859c2e96e9cde55f3797308a56a7bea08ff6fde6b52527e0ac3be855d80 WatchSource:0}: Error finding container 8cc6b859c2e96e9cde55f3797308a56a7bea08ff6fde6b52527e0ac3be855d80: Status 404 returned error can't find the container with id 8cc6b859c2e96e9cde55f3797308a56a7bea08ff6fde6b52527e0ac3be855d80 Jul 10 22:29:55.753552 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.753529 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-795cbfd498-4rrtw" Jul 10 22:29:55.766861 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.766834 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f586fdfd-crh7d" event={"ID":"0fae49b8-a6f3-41ab-bad5-753a8c39df6b","Type":"ContainerStarted","Data":"8cc6b859c2e96e9cde55f3797308a56a7bea08ff6fde6b52527e0ac3be855d80"} Jul 10 22:29:55.889968 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:55.889943 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-795cbfd498-4rrtw"] Jul 10 22:29:55.891777 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:29:55.891749 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a6e213_1eed_47bc_a5b3_8da897663feb.slice/crio-94cdbf22a3aa2a063b4f6ec8895d326f2a6a095f0975739984e6b435cb9df591 WatchSource:0}: Error finding container 94cdbf22a3aa2a063b4f6ec8895d326f2a6a095f0975739984e6b435cb9df591: Status 404 returned error can't find the container with id 94cdbf22a3aa2a063b4f6ec8895d326f2a6a095f0975739984e6b435cb9df591 Jul 10 22:29:56.776289 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:56.776214 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-795cbfd498-4rrtw" event={"ID":"a3a6e213-1eed-47bc-a5b3-8da897663feb","Type":"ContainerStarted","Data":"94cdbf22a3aa2a063b4f6ec8895d326f2a6a095f0975739984e6b435cb9df591"} Jul 10 22:29:58.054684 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:58.054575 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:58.142968 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:58.142936 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:29:58.787192 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:58.787154 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f586fdfd-crh7d" event={"ID":"0fae49b8-a6f3-41ab-bad5-753a8c39df6b","Type":"ContainerStarted","Data":"a5bd53a3cd1d981a87da50a2badc8b30b1b2c788612e43c0b2e0d4f81edfa909"} Jul 10 22:29:58.787394 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:58.787274 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f586fdfd-crh7d" Jul 10 22:29:58.788690 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:58.788668 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-795cbfd498-4rrtw" event={"ID":"a3a6e213-1eed-47bc-a5b3-8da897663feb","Type":"ContainerStarted","Data":"9bf4117703a8d22ac51b42a248d95ec46fa4120aefd435cfc97111a24539ff12"} Jul 10 22:29:58.788816 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:58.788758 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-795cbfd498-4rrtw" Jul 10 22:29:58.805710 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:58.805667 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f586fdfd-crh7d" podStartSLOduration=1.785960974 podStartE2EDuration="3.805654165s" totalImagesPullingTime="2.019693191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:29:58.804588631 +0000 UTC m=+462.637473016" watchObservedRunningTime="2026-07-10 22:29:58.805654165 +0000 UTC m=+462.638538549" Jul 10 22:29:58.825197 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:58.825150 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-795cbfd498-4rrtw" podStartSLOduration=1.970153957 podStartE2EDuration="3.825134142s" totalImagesPullingTime="1.854980185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:29:58.822666947 +0000 UTC m=+462.655551354" watchObservedRunningTime="2026-07-10 22:29:58.825134142 +0000 UTC m=+462.658018537" Jul 10 22:29:59.862235 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:29:59.862189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:30:09.799344 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:09.799310 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-795cbfd498-4rrtw" Jul 10 22:30:09.799733 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:09.799382 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f586fdfd-crh7d" Jul 10 22:30:09.859027 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:09.858994 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f586fdfd-crh7d"] Jul 10 22:30:09.859253 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:09.859202 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7f586fdfd-crh7d" podUID="0fae49b8-a6f3-41ab-bad5-753a8c39df6b" containerName="manager" containerID="cri-o://a5bd53a3cd1d981a87da50a2badc8b30b1b2c788612e43c0b2e0d4f81edfa909" gracePeriod=30 Jul 10 22:30:10.104587 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.104559 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f586fdfd-crh7d" Jul 10 22:30:10.273125 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.273082 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4tqh\" (UniqueName: \"kubernetes.io/projected/0fae49b8-a6f3-41ab-bad5-753a8c39df6b-kube-api-access-h4tqh\") pod \"0fae49b8-a6f3-41ab-bad5-753a8c39df6b\" (UID: \"0fae49b8-a6f3-41ab-bad5-753a8c39df6b\") " Jul 10 22:30:10.273125 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.273133 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fae49b8-a6f3-41ab-bad5-753a8c39df6b-webhook-cert\") pod \"0fae49b8-a6f3-41ab-bad5-753a8c39df6b\" (UID: \"0fae49b8-a6f3-41ab-bad5-753a8c39df6b\") " Jul 10 22:30:10.275358 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.275332 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fae49b8-a6f3-41ab-bad5-753a8c39df6b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0fae49b8-a6f3-41ab-bad5-753a8c39df6b" (UID: "0fae49b8-a6f3-41ab-bad5-753a8c39df6b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:30:10.275474 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.275353 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fae49b8-a6f3-41ab-bad5-753a8c39df6b-kube-api-access-h4tqh" (OuterVolumeSpecName: "kube-api-access-h4tqh") pod "0fae49b8-a6f3-41ab-bad5-753a8c39df6b" (UID: "0fae49b8-a6f3-41ab-bad5-753a8c39df6b"). InnerVolumeSpecName "kube-api-access-h4tqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:30:10.374154 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.374066 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4tqh\" (UniqueName: \"kubernetes.io/projected/0fae49b8-a6f3-41ab-bad5-753a8c39df6b-kube-api-access-h4tqh\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:30:10.374154 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.374103 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fae49b8-a6f3-41ab-bad5-753a8c39df6b-webhook-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:30:10.865002 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.864969 2565 generic.go:356] "Generic (PLEG): container finished" podID="0fae49b8-a6f3-41ab-bad5-753a8c39df6b" containerID="a5bd53a3cd1d981a87da50a2badc8b30b1b2c788612e43c0b2e0d4f81edfa909" exitCode=0 Jul 10 22:30:10.865388 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.865012 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f586fdfd-crh7d" Jul 10 22:30:10.865388 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.865057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f586fdfd-crh7d" event={"ID":"0fae49b8-a6f3-41ab-bad5-753a8c39df6b","Type":"ContainerDied","Data":"a5bd53a3cd1d981a87da50a2badc8b30b1b2c788612e43c0b2e0d4f81edfa909"} Jul 10 22:30:10.865388 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.865102 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f586fdfd-crh7d" event={"ID":"0fae49b8-a6f3-41ab-bad5-753a8c39df6b","Type":"ContainerDied","Data":"8cc6b859c2e96e9cde55f3797308a56a7bea08ff6fde6b52527e0ac3be855d80"} Jul 10 22:30:10.865388 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.865123 2565 scope.go:117] "RemoveContainer" containerID="a5bd53a3cd1d981a87da50a2badc8b30b1b2c788612e43c0b2e0d4f81edfa909" Jul 10 22:30:10.874073 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.874059 2565 scope.go:117] "RemoveContainer" containerID="a5bd53a3cd1d981a87da50a2badc8b30b1b2c788612e43c0b2e0d4f81edfa909" Jul 10 22:30:10.874309 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:30:10.874293 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5bd53a3cd1d981a87da50a2badc8b30b1b2c788612e43c0b2e0d4f81edfa909\": container with ID starting with a5bd53a3cd1d981a87da50a2badc8b30b1b2c788612e43c0b2e0d4f81edfa909 not found: ID does not exist" containerID="a5bd53a3cd1d981a87da50a2badc8b30b1b2c788612e43c0b2e0d4f81edfa909" Jul 10 22:30:10.874355 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.874318 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bd53a3cd1d981a87da50a2badc8b30b1b2c788612e43c0b2e0d4f81edfa909"} err="failed to get container status \"a5bd53a3cd1d981a87da50a2badc8b30b1b2c788612e43c0b2e0d4f81edfa909\": rpc error: code = NotFound desc = could not find container \"a5bd53a3cd1d981a87da50a2badc8b30b1b2c788612e43c0b2e0d4f81edfa909\": container with ID starting with a5bd53a3cd1d981a87da50a2badc8b30b1b2c788612e43c0b2e0d4f81edfa909 not found: ID does not exist" Jul 10 22:30:10.886339 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.886317 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f586fdfd-crh7d"] Jul 10 22:30:10.888273 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:10.888254 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f586fdfd-crh7d"] Jul 10 22:30:11.690420 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:11.690385 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fae49b8-a6f3-41ab-bad5-753a8c39df6b" path="/var/lib/kubelet/pods/0fae49b8-a6f3-41ab-bad5-753a8c39df6b/volumes" Jul 10 22:30:30.540825 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:30.540775 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:30:31.074265 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.074238 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:30:31.161900 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.161869 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg"] Jul 10 22:30:31.162200 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.162188 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fae49b8-a6f3-41ab-bad5-753a8c39df6b" containerName="manager" Jul 10 22:30:31.162244 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.162202 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fae49b8-a6f3-41ab-bad5-753a8c39df6b" containerName="manager" Jul 10 22:30:31.162289 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.162280 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fae49b8-a6f3-41ab-bad5-753a8c39df6b" containerName="manager" Jul 10 22:30:31.166906 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.166889 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:31.169478 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.169456 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 10 22:30:31.169607 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.169459 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2rct7\"" Jul 10 22:30:31.175217 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.175195 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg"] Jul 10 22:30:31.222434 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.222404 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dd1c61b-9a82-48b4-bab5-b6c015301d93-openshift-service-ca\") pod \"maas-api-65ccc6b695-xrkrg\" (UID: \"3dd1c61b-9a82-48b4-bab5-b6c015301d93\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:31.222600 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.222451 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdggd\" (UniqueName: \"kubernetes.io/projected/3dd1c61b-9a82-48b4-bab5-b6c015301d93-kube-api-access-rdggd\") pod \"maas-api-65ccc6b695-xrkrg\" (UID: \"3dd1c61b-9a82-48b4-bab5-b6c015301d93\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:31.222600 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.222489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dd1c61b-9a82-48b4-bab5-b6c015301d93-maas-api-tls\") pod \"maas-api-65ccc6b695-xrkrg\" (UID: \"3dd1c61b-9a82-48b4-bab5-b6c015301d93\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:31.323369 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.323332 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdggd\" (UniqueName: \"kubernetes.io/projected/3dd1c61b-9a82-48b4-bab5-b6c015301d93-kube-api-access-rdggd\") pod \"maas-api-65ccc6b695-xrkrg\" (UID: \"3dd1c61b-9a82-48b4-bab5-b6c015301d93\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:31.323552 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.323391 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dd1c61b-9a82-48b4-bab5-b6c015301d93-maas-api-tls\") pod \"maas-api-65ccc6b695-xrkrg\" (UID: \"3dd1c61b-9a82-48b4-bab5-b6c015301d93\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:31.323552 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.323449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dd1c61b-9a82-48b4-bab5-b6c015301d93-openshift-service-ca\") pod \"maas-api-65ccc6b695-xrkrg\" (UID: \"3dd1c61b-9a82-48b4-bab5-b6c015301d93\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:31.323552 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:30:31.323536 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 10 22:30:31.323665 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:30:31.323603 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dd1c61b-9a82-48b4-bab5-b6c015301d93-maas-api-tls podName:3dd1c61b-9a82-48b4-bab5-b6c015301d93 nodeName:}" failed. No retries permitted until 2026-07-10 22:30:31.823583184 +0000 UTC m=+495.656467548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3dd1c61b-9a82-48b4-bab5-b6c015301d93-maas-api-tls") pod "maas-api-65ccc6b695-xrkrg" (UID: "3dd1c61b-9a82-48b4-bab5-b6c015301d93") : secret "maas-api-serving-cert" not found Jul 10 22:30:31.324090 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.324073 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dd1c61b-9a82-48b4-bab5-b6c015301d93-openshift-service-ca\") pod \"maas-api-65ccc6b695-xrkrg\" (UID: \"3dd1c61b-9a82-48b4-bab5-b6c015301d93\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:31.332970 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.332899 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdggd\" (UniqueName: \"kubernetes.io/projected/3dd1c61b-9a82-48b4-bab5-b6c015301d93-kube-api-access-rdggd\") pod \"maas-api-65ccc6b695-xrkrg\" (UID: \"3dd1c61b-9a82-48b4-bab5-b6c015301d93\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:31.498105 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.498072 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:30:31.666508 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.666420 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:30:31.753369 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.753334 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:30:31.760293 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.760266 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-xqdf8"] Jul 10 22:30:31.766071 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.766050 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-xqdf8" Jul 10 22:30:31.768347 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.768329 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 22:30:31.771663 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.771639 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-xqdf8"] Jul 10 22:30:31.785632 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.785602 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-h7xq9"] Jul 10 22:30:31.789473 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.789455 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-h7xq9" Jul 10 22:30:31.791902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.791885 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cbtg6\"" Jul 10 22:30:31.796875 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.796854 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-h7xq9"] Jul 10 22:30:31.828244 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.828216 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9a29f3-8817-4929-bc2c-7d8be1c699b9-plugins-config-volume\") pod \"payload-processing-58f86c7798-h7xq9\" (UID: \"0f9a29f3-8817-4929-bc2c-7d8be1c699b9\") " pod="openshift-ingress/payload-processing-58f86c7798-h7xq9" Jul 10 22:30:31.828398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.828264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5vj4\" (UniqueName: \"kubernetes.io/projected/0f9a29f3-8817-4929-bc2c-7d8be1c699b9-kube-api-access-w5vj4\") pod \"payload-processing-58f86c7798-h7xq9\" (UID: \"0f9a29f3-8817-4929-bc2c-7d8be1c699b9\") " pod="openshift-ingress/payload-processing-58f86c7798-h7xq9" Jul 10 22:30:31.828398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.828293 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dd1c61b-9a82-48b4-bab5-b6c015301d93-maas-api-tls\") pod \"maas-api-65ccc6b695-xrkrg\" (UID: \"3dd1c61b-9a82-48b4-bab5-b6c015301d93\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:31.828472 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.828432 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q968\" (UniqueName: \"kubernetes.io/projected/3ff97a1d-ad63-41f6-8768-b90dcfb72795-kube-api-access-6q968\") pod \"payload-pre-processing-7964457696-xqdf8\" (UID: \"3ff97a1d-ad63-41f6-8768-b90dcfb72795\") " pod="openshift-ingress/payload-pre-processing-7964457696-xqdf8" Jul 10 22:30:31.828537 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.828520 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff97a1d-ad63-41f6-8768-b90dcfb72795-plugins-config-volume\") pod \"payload-pre-processing-7964457696-xqdf8\" (UID: \"3ff97a1d-ad63-41f6-8768-b90dcfb72795\") " pod="openshift-ingress/payload-pre-processing-7964457696-xqdf8" Jul 10 22:30:31.830694 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.830666 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dd1c61b-9a82-48b4-bab5-b6c015301d93-maas-api-tls\") pod \"maas-api-65ccc6b695-xrkrg\" (UID: \"3dd1c61b-9a82-48b4-bab5-b6c015301d93\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:31.929211 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.929129 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff97a1d-ad63-41f6-8768-b90dcfb72795-plugins-config-volume\") pod \"payload-pre-processing-7964457696-xqdf8\" (UID: \"3ff97a1d-ad63-41f6-8768-b90dcfb72795\") " pod="openshift-ingress/payload-pre-processing-7964457696-xqdf8" Jul 10 22:30:31.929211 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.929168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9a29f3-8817-4929-bc2c-7d8be1c699b9-plugins-config-volume\") pod \"payload-processing-58f86c7798-h7xq9\" (UID: \"0f9a29f3-8817-4929-bc2c-7d8be1c699b9\") " pod="openshift-ingress/payload-processing-58f86c7798-h7xq9" Jul 10 22:30:31.929441 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.929323 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5vj4\" (UniqueName: \"kubernetes.io/projected/0f9a29f3-8817-4929-bc2c-7d8be1c699b9-kube-api-access-w5vj4\") pod \"payload-processing-58f86c7798-h7xq9\" (UID: \"0f9a29f3-8817-4929-bc2c-7d8be1c699b9\") " pod="openshift-ingress/payload-processing-58f86c7798-h7xq9" Jul 10 22:30:31.929441 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.929390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q968\" (UniqueName: \"kubernetes.io/projected/3ff97a1d-ad63-41f6-8768-b90dcfb72795-kube-api-access-6q968\") pod \"payload-pre-processing-7964457696-xqdf8\" (UID: \"3ff97a1d-ad63-41f6-8768-b90dcfb72795\") " pod="openshift-ingress/payload-pre-processing-7964457696-xqdf8" Jul 10 22:30:31.929917 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.929895 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9a29f3-8817-4929-bc2c-7d8be1c699b9-plugins-config-volume\") pod \"payload-processing-58f86c7798-h7xq9\" (UID: \"0f9a29f3-8817-4929-bc2c-7d8be1c699b9\") " pod="openshift-ingress/payload-processing-58f86c7798-h7xq9" Jul 10 22:30:31.930019 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.929932 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff97a1d-ad63-41f6-8768-b90dcfb72795-plugins-config-volume\") pod \"payload-pre-processing-7964457696-xqdf8\" (UID: \"3ff97a1d-ad63-41f6-8768-b90dcfb72795\") " pod="openshift-ingress/payload-pre-processing-7964457696-xqdf8" Jul 10 22:30:31.937184 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.937153 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q968\" (UniqueName: \"kubernetes.io/projected/3ff97a1d-ad63-41f6-8768-b90dcfb72795-kube-api-access-6q968\") pod \"payload-pre-processing-7964457696-xqdf8\" (UID: \"3ff97a1d-ad63-41f6-8768-b90dcfb72795\") " pod="openshift-ingress/payload-pre-processing-7964457696-xqdf8" Jul 10 22:30:31.937387 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:31.937370 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5vj4\" (UniqueName: \"kubernetes.io/projected/0f9a29f3-8817-4929-bc2c-7d8be1c699b9-kube-api-access-w5vj4\") pod \"payload-processing-58f86c7798-h7xq9\" (UID: \"0f9a29f3-8817-4929-bc2c-7d8be1c699b9\") " pod="openshift-ingress/payload-processing-58f86c7798-h7xq9" Jul 10 22:30:32.077789 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:32.077745 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-xqdf8" Jul 10 22:30:32.078000 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:32.077935 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:32.100166 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:32.100138 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-h7xq9" Jul 10 22:30:32.235542 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:32.235444 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-xqdf8"] Jul 10 22:30:32.237960 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:30:32.237928 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff97a1d_ad63_41f6_8768_b90dcfb72795.slice/crio-86d05a050b4b8fe325a378a2286f2b02b6a25b6d35e30ddaa26cf7405e180083 WatchSource:0}: Error finding container 86d05a050b4b8fe325a378a2286f2b02b6a25b6d35e30ddaa26cf7405e180083: Status 404 returned error can't find the container with id 86d05a050b4b8fe325a378a2286f2b02b6a25b6d35e30ddaa26cf7405e180083 Jul 10 22:30:32.260167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:32.260136 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg"] Jul 10 22:30:32.261911 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:30:32.261881 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd1c61b_9a82_48b4_bab5_b6c015301d93.slice/crio-359c6657e84e5ddb4d296839bc3d528933bdedda007de02a5e55fd6e74ea2a85 WatchSource:0}: Error finding container 359c6657e84e5ddb4d296839bc3d528933bdedda007de02a5e55fd6e74ea2a85: Status 404 returned error can't find the container with id 359c6657e84e5ddb4d296839bc3d528933bdedda007de02a5e55fd6e74ea2a85 Jul 10 22:30:32.287159 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:32.287137 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-h7xq9"] Jul 10 22:30:32.288885 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:30:32.288859 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9a29f3_8817_4929_bc2c_7d8be1c699b9.slice/crio-3277e992c4c9d9ba566be1f3edc67033021ef86ce95f9a26ad0c0156e296daf9 WatchSource:0}: Error finding container 3277e992c4c9d9ba566be1f3edc67033021ef86ce95f9a26ad0c0156e296daf9: Status 404 returned error can't find the container with id 3277e992c4c9d9ba566be1f3edc67033021ef86ce95f9a26ad0c0156e296daf9 Jul 10 22:30:32.975755 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:32.975716 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-h7xq9" event={"ID":"0f9a29f3-8817-4929-bc2c-7d8be1c699b9","Type":"ContainerStarted","Data":"3277e992c4c9d9ba566be1f3edc67033021ef86ce95f9a26ad0c0156e296daf9"} Jul 10 22:30:32.977782 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:32.977751 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" event={"ID":"3dd1c61b-9a82-48b4-bab5-b6c015301d93","Type":"ContainerStarted","Data":"359c6657e84e5ddb4d296839bc3d528933bdedda007de02a5e55fd6e74ea2a85"} Jul 10 22:30:32.979488 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:32.979465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-xqdf8" event={"ID":"3ff97a1d-ad63-41f6-8768-b90dcfb72795","Type":"ContainerStarted","Data":"86d05a050b4b8fe325a378a2286f2b02b6a25b6d35e30ddaa26cf7405e180083"} Jul 10 22:30:34.992295 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:34.992257 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-h7xq9" event={"ID":"0f9a29f3-8817-4929-bc2c-7d8be1c699b9","Type":"ContainerStarted","Data":"e6a07eafb44ea99b39209425e4747d0d8e459ed300ba69c00796a9fd71b44351"} Jul 10 22:30:34.992755 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:34.992370 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-h7xq9" Jul 10 22:30:34.993817 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:34.993784 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" event={"ID":"3dd1c61b-9a82-48b4-bab5-b6c015301d93","Type":"ContainerStarted","Data":"b12156317e437557d13d66dc099427f53c1ccd420f00a2b790297868234b6955"} Jul 10 22:30:34.993918 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:34.993892 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:34.995319 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:34.995298 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-xqdf8" event={"ID":"3ff97a1d-ad63-41f6-8768-b90dcfb72795","Type":"ContainerStarted","Data":"fac8b740d68f904c83a4af9e2d11080c322c0701117ca0677b899f94f08300cc"} Jul 10 22:30:34.995453 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:34.995397 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-xqdf8" Jul 10 22:30:35.008057 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:35.008014 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-h7xq9" podStartSLOduration=1.929096683 podStartE2EDuration="4.008002247s" totalImagesPullingTime="2.078905564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:30:35.006786892 +0000 UTC m=+498.839671305" watchObservedRunningTime="2026-07-10 22:30:35.008002247 +0000 UTC m=+498.840886637" Jul 10 22:30:35.022650 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:35.022593 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" podStartSLOduration=1.9164673410000002 podStartE2EDuration="4.022573504s" totalImagesPullingTime="2.106106163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:30:35.021297821 +0000 UTC m=+498.854182204" watchObservedRunningTime="2026-07-10 22:30:35.022573504 +0000 UTC m=+498.855457888" Jul 10 22:30:40.024354 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:40.024321 2565 generic.go:356] "Generic (PLEG): container finished" podID="a688576f-a240-451c-84a0-31ef6c45f6a9" containerID="9d66b2c1b32a2e49ed1e7a0c293518d7c3317bce5a4c474d27b054de870df62f" exitCode=0 Jul 10 22:30:40.024823 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:40.024402 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" event={"ID":"a688576f-a240-451c-84a0-31ef6c45f6a9","Type":"ContainerDied","Data":"9d66b2c1b32a2e49ed1e7a0c293518d7c3317bce5a4c474d27b054de870df62f"} Jul 10 22:30:40.024879 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:40.024854 2565 scope.go:117] "RemoveContainer" containerID="9d66b2c1b32a2e49ed1e7a0c293518d7c3317bce5a4c474d27b054de870df62f" Jul 10 22:30:40.026022 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:40.025981 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:30:40.026411 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:40.026073 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:30:40.026411 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:40.026118 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:30:40.047442 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:40.047377 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-xqdf8" podStartSLOduration=6.915043114 podStartE2EDuration="9.047354431s" totalImagesPullingTime="2.132311317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:30:35.038606096 +0000 UTC m=+498.871490474" watchObservedRunningTime="2026-07-10 22:30:40.047354431 +0000 UTC m=+503.880239009" Jul 10 22:30:41.006710 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:41.006684 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-xrkrg" Jul 10 22:30:41.038063 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:41.038005 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" event={"ID":"a688576f-a240-451c-84a0-31ef6c45f6a9","Type":"ContainerStarted","Data":"aa4345f5e0059e29b9ab4586718f2e25119cbfd661c75a669aadec3256c5aee1"} Jul 10 22:30:41.972862 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:41.972826 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:30:41.997666 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:41.997633 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:30:41.998842 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:41.998817 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sgnbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 10 22:30:41.998956 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:41.998874 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" podUID="a688576f-a240-451c-84a0-31ef6c45f6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 10 22:30:42.079269 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:42.079230 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:30:42.998071 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:42.998034 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sgnbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 10 22:30:42.998276 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:42.998107 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" podUID="a688576f-a240-451c-84a0-31ef6c45f6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 10 22:30:43.893246 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:43.893189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:30:43.991080 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:43.991044 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:30:43.998110 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:43.998088 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sgnbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 10 22:30:43.998190 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:43.998147 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" podUID="a688576f-a240-451c-84a0-31ef6c45f6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 10 22:30:45.021188 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:45.021157 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:30:45.021582 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:45.021430 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:30:45.022122 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:45.022107 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sgnbx" Jul 10 22:30:46.003406 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:46.003368 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-h7xq9" Jul 10 22:30:46.003590 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:46.003477 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-xqdf8" Jul 10 22:30:52.877279 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.877245 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5"] Jul 10 22:30:52.880082 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.880057 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:52.882449 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.882423 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 22:30:52.883315 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.883283 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 22:30:52.883465 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.883314 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-szs6p\"" Jul 10 22:30:52.883465 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.883319 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 10 22:30:52.890441 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.890417 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5"] Jul 10 22:30:52.903784 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.903752 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:30:52.913320 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.913292 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493d0dc5-3aa2-455a-b4b7-efb46438745b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:52.913483 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.913348 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493d0dc5-3aa2-455a-b4b7-efb46438745b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:52.913483 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.913432 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493d0dc5-3aa2-455a-b4b7-efb46438745b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:52.913585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.913486 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493d0dc5-3aa2-455a-b4b7-efb46438745b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:52.913585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.913535 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkb8x\" (UniqueName: \"kubernetes.io/projected/493d0dc5-3aa2-455a-b4b7-efb46438745b-kube-api-access-jkb8x\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:52.913585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:52.913568 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493d0dc5-3aa2-455a-b4b7-efb46438745b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.014314 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.014272 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493d0dc5-3aa2-455a-b4b7-efb46438745b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.014508 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.014334 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkb8x\" (UniqueName: \"kubernetes.io/projected/493d0dc5-3aa2-455a-b4b7-efb46438745b-kube-api-access-jkb8x\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.014508 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.014355 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493d0dc5-3aa2-455a-b4b7-efb46438745b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.014508 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.014379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493d0dc5-3aa2-455a-b4b7-efb46438745b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.014508 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.014409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493d0dc5-3aa2-455a-b4b7-efb46438745b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.014508 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.014441 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493d0dc5-3aa2-455a-b4b7-efb46438745b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.014884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.014736 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493d0dc5-3aa2-455a-b4b7-efb46438745b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.014884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.014785 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493d0dc5-3aa2-455a-b4b7-efb46438745b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.014884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.014857 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493d0dc5-3aa2-455a-b4b7-efb46438745b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.016807 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.016769 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493d0dc5-3aa2-455a-b4b7-efb46438745b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.017116 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.017101 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493d0dc5-3aa2-455a-b4b7-efb46438745b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.021923 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.021897 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkb8x\" (UniqueName: \"kubernetes.io/projected/493d0dc5-3aa2-455a-b4b7-efb46438745b-kube-api-access-jkb8x\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5\" (UID: \"493d0dc5-3aa2-455a-b4b7-efb46438745b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.192384 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.192294 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:30:53.330472 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:53.330441 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5"] Jul 10 22:30:53.331966 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:30:53.331937 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493d0dc5_3aa2_455a_b4b7_efb46438745b.slice/crio-627dade7885aab4ce51097bb610e627eb7889e42589e48f76ebb9e2e25237150 WatchSource:0}: Error finding container 627dade7885aab4ce51097bb610e627eb7889e42589e48f76ebb9e2e25237150: Status 404 returned error can't find the container with id 627dade7885aab4ce51097bb610e627eb7889e42589e48f76ebb9e2e25237150 Jul 10 22:30:54.111493 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:54.111460 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" event={"ID":"493d0dc5-3aa2-455a-b4b7-efb46438745b","Type":"ContainerStarted","Data":"627dade7885aab4ce51097bb610e627eb7889e42589e48f76ebb9e2e25237150"} Jul 10 22:30:55.118043 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:55.118004 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" event={"ID":"493d0dc5-3aa2-455a-b4b7-efb46438745b","Type":"ContainerStarted","Data":"176412c5e3491f02151994a20afdd9e449d16431461549b8fe066cd68d0738e8"} Jul 10 22:30:55.135996 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:55.135946 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" podStartSLOduration=2.016893991 podStartE2EDuration="3.135928631s" totalImagesPullingTime="1.11903464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:30:55.134331699 +0000 UTC m=+518.967216088" watchObservedRunningTime="2026-07-10 22:30:55.135928631 +0000 UTC m=+518.968813066" Jul 10 22:30:59.117279 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:30:59.117234 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:31:03.193110 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:31:03.193074 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:31:03.193605 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:31:03.193130 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:31:03.205757 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:31:03.205723 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:31:04.174256 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:31:04.174221 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5" Jul 10 22:31:06.917694 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:31:06.917661 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:31:17.632124 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:31:17.632044 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:31:23.713356 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:31:23.713316 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:31:28.640717 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:31:28.640678 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:31:34.413702 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:31:34.413652 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:31:41.812481 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:31:41.812434 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:32:18.214285 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:32:18.214254 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:32:18.215574 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:32:18.215551 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:32:21.365960 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:32:21.365906 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:32:23.051709 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:32:23.051666 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:32:40.899898 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:32:40.899784 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:32:40.952533 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:32:40.952496 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:33:32.859707 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:32.859672 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:33:32.958085 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:32.958052 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-795cbfd498-4rrtw"] Jul 10 22:33:32.958286 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:32.958262 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-795cbfd498-4rrtw" podUID="a3a6e213-1eed-47bc-a5b3-8da897663feb" containerName="manager" containerID="cri-o://9bf4117703a8d22ac51b42a248d95ec46fa4120aefd435cfc97111a24539ff12" gracePeriod=30 Jul 10 22:33:33.213046 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.213023 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-795cbfd498-4rrtw" Jul 10 22:33:33.270515 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.270482 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkdnl\" (UniqueName: \"kubernetes.io/projected/a3a6e213-1eed-47bc-a5b3-8da897663feb-kube-api-access-bkdnl\") pod \"a3a6e213-1eed-47bc-a5b3-8da897663feb\" (UID: \"a3a6e213-1eed-47bc-a5b3-8da897663feb\") " Jul 10 22:33:33.270707 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.270540 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3a6e213-1eed-47bc-a5b3-8da897663feb-webhook-cert\") pod \"a3a6e213-1eed-47bc-a5b3-8da897663feb\" (UID: \"a3a6e213-1eed-47bc-a5b3-8da897663feb\") " Jul 10 22:33:33.272593 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.272560 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a6e213-1eed-47bc-a5b3-8da897663feb-kube-api-access-bkdnl" (OuterVolumeSpecName: "kube-api-access-bkdnl") pod "a3a6e213-1eed-47bc-a5b3-8da897663feb" (UID: "a3a6e213-1eed-47bc-a5b3-8da897663feb"). InnerVolumeSpecName "kube-api-access-bkdnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:33:33.272710 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.272595 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a6e213-1eed-47bc-a5b3-8da897663feb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a3a6e213-1eed-47bc-a5b3-8da897663feb" (UID: "a3a6e213-1eed-47bc-a5b3-8da897663feb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:33:33.371181 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.371143 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3a6e213-1eed-47bc-a5b3-8da897663feb-webhook-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:33:33.371181 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.371174 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkdnl\" (UniqueName: \"kubernetes.io/projected/a3a6e213-1eed-47bc-a5b3-8da897663feb-kube-api-access-bkdnl\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:33:33.919579 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.919539 2565 generic.go:356] "Generic (PLEG): container finished" podID="a3a6e213-1eed-47bc-a5b3-8da897663feb" containerID="9bf4117703a8d22ac51b42a248d95ec46fa4120aefd435cfc97111a24539ff12" exitCode=0 Jul 10 22:33:33.920015 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.919588 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-795cbfd498-4rrtw" Jul 10 22:33:33.920015 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.919627 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-795cbfd498-4rrtw" event={"ID":"a3a6e213-1eed-47bc-a5b3-8da897663feb","Type":"ContainerDied","Data":"9bf4117703a8d22ac51b42a248d95ec46fa4120aefd435cfc97111a24539ff12"} Jul 10 22:33:33.920015 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.919683 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-795cbfd498-4rrtw" event={"ID":"a3a6e213-1eed-47bc-a5b3-8da897663feb","Type":"ContainerDied","Data":"94cdbf22a3aa2a063b4f6ec8895d326f2a6a095f0975739984e6b435cb9df591"} Jul 10 22:33:33.920015 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.919713 2565 scope.go:117] "RemoveContainer" containerID="9bf4117703a8d22ac51b42a248d95ec46fa4120aefd435cfc97111a24539ff12" Jul 10 22:33:33.928547 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.928529 2565 scope.go:117] "RemoveContainer" containerID="9bf4117703a8d22ac51b42a248d95ec46fa4120aefd435cfc97111a24539ff12" Jul 10 22:33:33.928822 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:33:33.928788 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf4117703a8d22ac51b42a248d95ec46fa4120aefd435cfc97111a24539ff12\": container with ID starting with 9bf4117703a8d22ac51b42a248d95ec46fa4120aefd435cfc97111a24539ff12 not found: ID does not exist" containerID="9bf4117703a8d22ac51b42a248d95ec46fa4120aefd435cfc97111a24539ff12" Jul 10 22:33:33.928886 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.928829 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf4117703a8d22ac51b42a248d95ec46fa4120aefd435cfc97111a24539ff12"} err="failed to get container status \"9bf4117703a8d22ac51b42a248d95ec46fa4120aefd435cfc97111a24539ff12\": rpc error: code = NotFound desc = could not find container \"9bf4117703a8d22ac51b42a248d95ec46fa4120aefd435cfc97111a24539ff12\": container with ID starting with 9bf4117703a8d22ac51b42a248d95ec46fa4120aefd435cfc97111a24539ff12 not found: ID does not exist" Jul 10 22:33:33.935722 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.935698 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-795cbfd498-4rrtw"] Jul 10 22:33:33.939836 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:33.939814 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-795cbfd498-4rrtw"] Jul 10 22:33:34.596635 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.596597 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:33:34.708728 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.708684 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-795cbfd498-wtkgt"] Jul 10 22:33:34.709350 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.709318 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3a6e213-1eed-47bc-a5b3-8da897663feb" containerName="manager" Jul 10 22:33:34.709350 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.709341 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3a6e213-1eed-47bc-a5b3-8da897663feb" containerName="manager" Jul 10 22:33:34.709517 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.709414 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3a6e213-1eed-47bc-a5b3-8da897663feb" containerName="manager" Jul 10 22:33:34.713259 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.713243 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-795cbfd498-wtkgt" Jul 10 22:33:34.716482 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.716456 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 22:33:34.716639 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.716520 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-n68dc\"" Jul 10 22:33:34.716639 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.716530 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 22:33:34.722621 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.722595 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-795cbfd498-wtkgt"] Jul 10 22:33:34.784025 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.783984 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba17e396-b47f-494b-9efd-057aa65a379a-webhook-cert\") pod \"maas-controller-795cbfd498-wtkgt\" (UID: \"ba17e396-b47f-494b-9efd-057aa65a379a\") " pod="opendatahub/maas-controller-795cbfd498-wtkgt" Jul 10 22:33:34.784215 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.784036 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9p2d\" (UniqueName: \"kubernetes.io/projected/ba17e396-b47f-494b-9efd-057aa65a379a-kube-api-access-t9p2d\") pod \"maas-controller-795cbfd498-wtkgt\" (UID: \"ba17e396-b47f-494b-9efd-057aa65a379a\") " pod="opendatahub/maas-controller-795cbfd498-wtkgt" Jul 10 22:33:34.884921 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.884821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9p2d\" (UniqueName: \"kubernetes.io/projected/ba17e396-b47f-494b-9efd-057aa65a379a-kube-api-access-t9p2d\") pod \"maas-controller-795cbfd498-wtkgt\" (UID: \"ba17e396-b47f-494b-9efd-057aa65a379a\") " pod="opendatahub/maas-controller-795cbfd498-wtkgt" Jul 10 22:33:34.885110 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.884940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba17e396-b47f-494b-9efd-057aa65a379a-webhook-cert\") pod \"maas-controller-795cbfd498-wtkgt\" (UID: \"ba17e396-b47f-494b-9efd-057aa65a379a\") " pod="opendatahub/maas-controller-795cbfd498-wtkgt" Jul 10 22:33:34.887419 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.887394 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba17e396-b47f-494b-9efd-057aa65a379a-webhook-cert\") pod \"maas-controller-795cbfd498-wtkgt\" (UID: \"ba17e396-b47f-494b-9efd-057aa65a379a\") " pod="opendatahub/maas-controller-795cbfd498-wtkgt" Jul 10 22:33:34.893566 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:34.893541 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9p2d\" (UniqueName: \"kubernetes.io/projected/ba17e396-b47f-494b-9efd-057aa65a379a-kube-api-access-t9p2d\") pod \"maas-controller-795cbfd498-wtkgt\" (UID: \"ba17e396-b47f-494b-9efd-057aa65a379a\") " pod="opendatahub/maas-controller-795cbfd498-wtkgt" Jul 10 22:33:35.025036 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:35.024996 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-795cbfd498-wtkgt" Jul 10 22:33:35.161732 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:35.161704 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-795cbfd498-wtkgt"] Jul 10 22:33:35.163577 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:33:35.163548 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba17e396_b47f_494b_9efd_057aa65a379a.slice/crio-1e433aeec3195e8ee32a6efecf8d8cdb6ebe6766119d2a2614ad88115eebd021 WatchSource:0}: Error finding container 1e433aeec3195e8ee32a6efecf8d8cdb6ebe6766119d2a2614ad88115eebd021: Status 404 returned error can't find the container with id 1e433aeec3195e8ee32a6efecf8d8cdb6ebe6766119d2a2614ad88115eebd021 Jul 10 22:33:35.165079 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:35.165058 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 22:33:35.691525 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:35.691499 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3a6e213-1eed-47bc-a5b3-8da897663feb" path="/var/lib/kubelet/pods/a3a6e213-1eed-47bc-a5b3-8da897663feb/volumes" Jul 10 22:33:35.933657 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:35.933621 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-795cbfd498-wtkgt" event={"ID":"ba17e396-b47f-494b-9efd-057aa65a379a","Type":"ContainerStarted","Data":"0a71afc0e24887dd7519838121254a87bcee986c998f25844fdfe3c033e0ee38"} Jul 10 22:33:35.933861 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:35.933674 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-795cbfd498-wtkgt" event={"ID":"ba17e396-b47f-494b-9efd-057aa65a379a","Type":"ContainerStarted","Data":"1e433aeec3195e8ee32a6efecf8d8cdb6ebe6766119d2a2614ad88115eebd021"} Jul 10 22:33:35.933861 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:35.933755 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-795cbfd498-wtkgt" Jul 10 22:33:35.950635 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:35.950545 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-795cbfd498-wtkgt" podStartSLOduration=1.4246461639999999 podStartE2EDuration="1.950529067s" totalImagesPullingTime="525.882903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:33:35.948375092 +0000 UTC m=+679.781259488" watchObservedRunningTime="2026-07-10 22:33:35.950529067 +0000 UTC m=+679.783413449" Jul 10 22:33:44.115654 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:44.115620 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:33:44.166707 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:44.166678 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:33:46.944718 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:33:46.944682 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-795cbfd498-wtkgt" Jul 10 22:34:15.201957 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:34:15.201881 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:34:15.304269 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:34:15.304234 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:34:15.456176 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:34:15.456096 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:34:21.003876 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:34:21.003836 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:34:21.123361 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:34:21.123324 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:34:21.274167 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:34:21.274072 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:34:21.363330 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:34:21.363240 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:34:27.055641 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:34:27.055587 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:34:27.155532 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:34:27.155488 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:34:29.914618 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:34:29.914580 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:37:18.249133 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:37:18.249104 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:37:18.250656 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:37:18.250633 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:38:43.378122 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:38:43.378042 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:38:44.467813 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:38:44.467770 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:38:54.818220 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:38:54.818182 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:39:19.855405 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:39:19.855361 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:40:42.212503 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:40:42.212468 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:40:42.313433 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:40:42.313396 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:42:18.279748 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:42:18.279719 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:42:18.282980 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:42:18.282959 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:45:00.179961 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:00.179876 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m"] Jul 10 22:45:00.183189 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:00.183171 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m" Jul 10 22:45:00.186479 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:00.186456 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-khng7\"" Jul 10 22:45:00.193013 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:00.192978 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m" Jul 10 22:45:00.211676 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:00.211641 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m"] Jul 10 22:45:00.339906 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:00.339877 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m"] Jul 10 22:45:00.342467 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:45:00.342437 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899bf4f5_d0b5_4af8_b2a5_f46a5c4299ab.slice/crio-d5327056adc135d15c1295d2d64ef6c14d733215e73300636e4d08d5f6c9b092 WatchSource:0}: Error finding container d5327056adc135d15c1295d2d64ef6c14d733215e73300636e4d08d5f6c9b092: Status 404 returned error can't find the container with id d5327056adc135d15c1295d2d64ef6c14d733215e73300636e4d08d5f6c9b092 Jul 10 22:45:00.344303 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:00.344288 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 22:45:00.455896 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:00.455786 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m" event={"ID":"899bf4f5-d0b5-4af8-b2a5-f46a5c4299ab","Type":"ContainerStarted","Data":"d5327056adc135d15c1295d2d64ef6c14d733215e73300636e4d08d5f6c9b092"} Jul 10 22:45:03.476491 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:03.476457 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m" event={"ID":"899bf4f5-d0b5-4af8-b2a5-f46a5c4299ab","Type":"ContainerStarted","Data":"9359cebbcd7d000ce66bf5224717ad4d72234ab4545d8d07166d97e40ef66259"} Jul 10 22:45:03.490818 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:03.490752 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m" podStartSLOduration=1.145585985 podStartE2EDuration="3.49073638s" totalImagesPullingTime="2.345150395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:45:03.489541646 +0000 UTC m=+1367.322426029" watchObservedRunningTime="2026-07-10 22:45:03.49073638 +0000 UTC m=+1367.323620800" Jul 10 22:45:19.652084 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:19.652043 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:45:33.536351 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:33.536300 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:45:41.936211 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:41.936159 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:45:47.104327 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:45:47.104290 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:46:00.009869 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:00.009833 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m"] Jul 10 22:46:00.010255 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:00.010037 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m" podUID="899bf4f5-d0b5-4af8-b2a5-f46a5c4299ab" containerName="cleanup" containerID="cri-o://9359cebbcd7d000ce66bf5224717ad4d72234ab4545d8d07166d97e40ef66259" gracePeriod=30 Jul 10 22:46:00.456902 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:00.456782 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:46:12.445869 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:12.445826 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:46:30.191382 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:30.191350 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m" Jul 10 22:46:30.936299 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:30.936258 2565 generic.go:356] "Generic (PLEG): container finished" podID="899bf4f5-d0b5-4af8-b2a5-f46a5c4299ab" containerID="9359cebbcd7d000ce66bf5224717ad4d72234ab4545d8d07166d97e40ef66259" exitCode=137 Jul 10 22:46:30.936299 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:30.936302 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m" Jul 10 22:46:30.936513 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:30.936330 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m" event={"ID":"899bf4f5-d0b5-4af8-b2a5-f46a5c4299ab","Type":"ContainerDied","Data":"9359cebbcd7d000ce66bf5224717ad4d72234ab4545d8d07166d97e40ef66259"} Jul 10 22:46:30.936513 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:30.936384 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m" event={"ID":"899bf4f5-d0b5-4af8-b2a5-f46a5c4299ab","Type":"ContainerDied","Data":"d5327056adc135d15c1295d2d64ef6c14d733215e73300636e4d08d5f6c9b092"} Jul 10 22:46:30.936513 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:30.936413 2565 scope.go:117] "RemoveContainer" containerID="9359cebbcd7d000ce66bf5224717ad4d72234ab4545d8d07166d97e40ef66259" Jul 10 22:46:30.945454 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:30.945437 2565 scope.go:117] "RemoveContainer" containerID="9359cebbcd7d000ce66bf5224717ad4d72234ab4545d8d07166d97e40ef66259" Jul 10 22:46:30.945730 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:46:30.945710 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9359cebbcd7d000ce66bf5224717ad4d72234ab4545d8d07166d97e40ef66259\": container with ID starting with 9359cebbcd7d000ce66bf5224717ad4d72234ab4545d8d07166d97e40ef66259 not found: ID does not exist" containerID="9359cebbcd7d000ce66bf5224717ad4d72234ab4545d8d07166d97e40ef66259" Jul 10 22:46:30.945826 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:30.945741 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9359cebbcd7d000ce66bf5224717ad4d72234ab4545d8d07166d97e40ef66259"} err="failed to get container status \"9359cebbcd7d000ce66bf5224717ad4d72234ab4545d8d07166d97e40ef66259\": rpc error: code = NotFound desc = could not find container \"9359cebbcd7d000ce66bf5224717ad4d72234ab4545d8d07166d97e40ef66259\": container with ID starting with 9359cebbcd7d000ce66bf5224717ad4d72234ab4545d8d07166d97e40ef66259 not found: ID does not exist" Jul 10 22:46:30.956417 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:30.956390 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m"] Jul 10 22:46:30.958525 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:30.958505 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728725-ngq4m"] Jul 10 22:46:31.690850 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:46:31.690819 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="899bf4f5-d0b5-4af8-b2a5-f46a5c4299ab" path="/var/lib/kubelet/pods/899bf4f5-d0b5-4af8-b2a5-f46a5c4299ab/volumes" Jul 10 22:47:18.310719 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:47:18.310617 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:47:18.315288 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:47:18.315265 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:50:28.583914 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:50:28.583876 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:50:29.732433 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:50:29.732399 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:51:58.817268 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:51:58.817230 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:51:58.915904 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:51:58.915866 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:51:59.281348 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:51:59.281309 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:05.102743 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.102712 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:05.158387 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.158355 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:05.271339 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.271312 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb"] Jul 10 22:52:05.271703 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.271690 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="899bf4f5-d0b5-4af8-b2a5-f46a5c4299ab" containerName="cleanup" Jul 10 22:52:05.271745 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.271704 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="899bf4f5-d0b5-4af8-b2a5-f46a5c4299ab" containerName="cleanup" Jul 10 22:52:05.271841 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.271772 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="899bf4f5-d0b5-4af8-b2a5-f46a5c4299ab" containerName="cleanup" Jul 10 22:52:05.274751 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.274733 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.276939 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.276917 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2ace46a5-openshift-default-dockercfg-ptjk7\"" Jul 10 22:52:05.303100 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.303068 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb"] Jul 10 22:52:05.305015 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.304992 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:05.407648 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.407566 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:05.435529 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.435492 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-workload-certs\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.435690 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.435550 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv8f9\" (UniqueName: \"kubernetes.io/projected/caaa290f-bdc0-4f3e-a2af-d99890e497b2-kube-api-access-rv8f9\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.435690 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.435577 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-token\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.435690 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.435604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-envoy\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.435690 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.435621 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-workload-socket\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.435843 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.435726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-podinfo\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.435843 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.435758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-credential-socket\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.435843 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.435776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istiod-ca-cert\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.435940 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.435843 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-data\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.537158 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.537119 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-data\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.537158 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.537159 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-workload-certs\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.537378 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.537198 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv8f9\" (UniqueName: \"kubernetes.io/projected/caaa290f-bdc0-4f3e-a2af-d99890e497b2-kube-api-access-rv8f9\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.537378 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.537215 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-token\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.537378 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.537241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-envoy\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.537378 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.537258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-workload-socket\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.537378 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.537327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-podinfo\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.537378 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.537365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-credential-socket\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.537677 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.537390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istiod-ca-cert\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.537677 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.537544 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-data\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.537677 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.537609 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-workload-certs\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.537887 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.537709 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-credential-socket\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.537887 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.537774 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-workload-socket\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.538196 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.538172 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istiod-ca-cert\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.539622 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.539603 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-envoy\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.539972 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.539951 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-podinfo\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.546700 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.546676 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-token\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.546820 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.546738 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv8f9\" (UniqueName: \"kubernetes.io/projected/caaa290f-bdc0-4f3e-a2af-d99890e497b2-kube-api-access-rv8f9\") pod \"e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.585313 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.585284 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:05.725210 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.725172 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb"] Jul 10 22:52:05.728480 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:52:05.728439 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaaa290f_bdc0_4f3e_a2af_d99890e497b2.slice/crio-021417c01b99529abc5428632ab77c5fa16b06d163a522a5603770d83e243dc7 WatchSource:0}: Error finding container 021417c01b99529abc5428632ab77c5fa16b06d163a522a5603770d83e243dc7: Status 404 returned error can't find the container with id 021417c01b99529abc5428632ab77c5fa16b06d163a522a5603770d83e243dc7 Jul 10 22:52:05.730587 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.730553 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:52:05.730668 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.730625 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:52:05.730668 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:05.730658 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:52:06.403042 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:06.403009 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:06.501681 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:06.501648 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:06.671249 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:06.671160 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" event={"ID":"caaa290f-bdc0-4f3e-a2af-d99890e497b2","Type":"ContainerStarted","Data":"ab2f636084ca233f0d2533ebbdef39359b533b4d9d6e57f3990093c9a81eee6b"} Jul 10 22:52:06.671249 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:06.671225 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" event={"ID":"caaa290f-bdc0-4f3e-a2af-d99890e497b2","Type":"ContainerStarted","Data":"021417c01b99529abc5428632ab77c5fa16b06d163a522a5603770d83e243dc7"} Jul 10 22:52:06.691016 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:06.690968 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" podStartSLOduration=1.690953046 podStartE2EDuration="1.690953046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:52:06.689629269 +0000 UTC m=+1790.522513653" watchObservedRunningTime="2026-07-10 22:52:06.690953046 +0000 UTC m=+1790.523837429" Jul 10 22:52:07.585478 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:07.585435 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:07.590311 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:07.590287 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:07.676987 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:07.676956 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:07.678029 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:07.678009 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:08.872581 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:08.872548 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:11.495938 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:11.495903 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:11.653398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:11.653359 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb"] Jul 10 22:52:11.653576 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:11.653544 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" podUID="caaa290f-bdc0-4f3e-a2af-d99890e497b2" containerName="istio-proxy" containerID="cri-o://ab2f636084ca233f0d2533ebbdef39359b533b4d9d6e57f3990093c9a81eee6b" gracePeriod=30 Jul 10 22:52:11.678958 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:11.678923 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:11.848087 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:11.848003 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:13.707174 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:13.707139 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:16.908217 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.908191 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:16.939287 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.939205 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-envoy\") pod \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " Jul 10 22:52:16.939287 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.939267 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istiod-ca-cert\") pod \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " Jul 10 22:52:16.939492 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.939299 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-credential-socket\") pod \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " Jul 10 22:52:16.939492 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.939324 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-token\") pod \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " Jul 10 22:52:16.939492 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.939362 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-data\") pod \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " Jul 10 22:52:16.939492 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.939390 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-workload-certs\") pod \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " Jul 10 22:52:16.939715 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.939628 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-data" (OuterVolumeSpecName: "istio-data") pod "caaa290f-bdc0-4f3e-a2af-d99890e497b2" (UID: "caaa290f-bdc0-4f3e-a2af-d99890e497b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:16.939715 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.939640 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "caaa290f-bdc0-4f3e-a2af-d99890e497b2" (UID: "caaa290f-bdc0-4f3e-a2af-d99890e497b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:16.939715 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.939668 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "caaa290f-bdc0-4f3e-a2af-d99890e497b2" (UID: "caaa290f-bdc0-4f3e-a2af-d99890e497b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:52:16.939715 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.939675 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-podinfo\") pod \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " Jul 10 22:52:16.939993 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.939744 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv8f9\" (UniqueName: \"kubernetes.io/projected/caaa290f-bdc0-4f3e-a2af-d99890e497b2-kube-api-access-rv8f9\") pod \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " Jul 10 22:52:16.939993 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.939782 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-workload-socket\") pod \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\" (UID: \"caaa290f-bdc0-4f3e-a2af-d99890e497b2\") " Jul 10 22:52:16.940103 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.939993 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "caaa290f-bdc0-4f3e-a2af-d99890e497b2" (UID: "caaa290f-bdc0-4f3e-a2af-d99890e497b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:16.940204 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.940184 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "caaa290f-bdc0-4f3e-a2af-d99890e497b2" (UID: "caaa290f-bdc0-4f3e-a2af-d99890e497b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:16.940264 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.940215 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istiod-ca-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:16.940264 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.940236 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-credential-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:16.940264 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.940254 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-data\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:16.940431 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.940268 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-workload-certs\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:16.941504 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.941484 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "caaa290f-bdc0-4f3e-a2af-d99890e497b2" (UID: "caaa290f-bdc0-4f3e-a2af-d99890e497b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:16.941715 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.941696 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-token" (OuterVolumeSpecName: "istio-token") pod "caaa290f-bdc0-4f3e-a2af-d99890e497b2" (UID: "caaa290f-bdc0-4f3e-a2af-d99890e497b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:52:16.942188 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.942160 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caaa290f-bdc0-4f3e-a2af-d99890e497b2-kube-api-access-rv8f9" (OuterVolumeSpecName: "kube-api-access-rv8f9") pod "caaa290f-bdc0-4f3e-a2af-d99890e497b2" (UID: "caaa290f-bdc0-4f3e-a2af-d99890e497b2"). InnerVolumeSpecName "kube-api-access-rv8f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:52:16.942269 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:16.942174 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "caaa290f-bdc0-4f3e-a2af-d99890e497b2" (UID: "caaa290f-bdc0-4f3e-a2af-d99890e497b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 22:52:17.041371 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.041333 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-workload-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:17.041371 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.041368 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-envoy\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:17.041371 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.041379 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-token\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:17.041623 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.041387 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaa290f-bdc0-4f3e-a2af-d99890e497b2-istio-podinfo\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:17.041623 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.041395 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv8f9\" (UniqueName: \"kubernetes.io/projected/caaa290f-bdc0-4f3e-a2af-d99890e497b2-kube-api-access-rv8f9\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:17.494505 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.494473 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:17.596695 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.596665 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:17.673205 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.673176 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t"] Jul 10 22:52:17.673584 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.673569 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caaa290f-bdc0-4f3e-a2af-d99890e497b2" containerName="istio-proxy" Jul 10 22:52:17.673628 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.673585 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="caaa290f-bdc0-4f3e-a2af-d99890e497b2" containerName="istio-proxy" Jul 10 22:52:17.673668 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.673652 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="caaa290f-bdc0-4f3e-a2af-d99890e497b2" containerName="istio-proxy" Jul 10 22:52:17.677030 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.677009 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.679400 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.679377 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f617a605-openshift-default-dockercfg-d4vx9\"" Jul 10 22:52:17.694509 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.694482 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t"] Jul 10 22:52:17.716495 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.716468 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:17.734331 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.734300 2565 generic.go:356] "Generic (PLEG): container finished" podID="caaa290f-bdc0-4f3e-a2af-d99890e497b2" containerID="ab2f636084ca233f0d2533ebbdef39359b533b4d9d6e57f3990093c9a81eee6b" exitCode=0 Jul 10 22:52:17.734488 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.734345 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" Jul 10 22:52:17.734488 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.734384 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" event={"ID":"caaa290f-bdc0-4f3e-a2af-d99890e497b2","Type":"ContainerDied","Data":"ab2f636084ca233f0d2533ebbdef39359b533b4d9d6e57f3990093c9a81eee6b"} Jul 10 22:52:17.734488 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.734433 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb" event={"ID":"caaa290f-bdc0-4f3e-a2af-d99890e497b2","Type":"ContainerDied","Data":"021417c01b99529abc5428632ab77c5fa16b06d163a522a5603770d83e243dc7"} Jul 10 22:52:17.734488 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.734457 2565 scope.go:117] "RemoveContainer" containerID="ab2f636084ca233f0d2533ebbdef39359b533b4d9d6e57f3990093c9a81eee6b" Jul 10 22:52:17.743671 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.743651 2565 scope.go:117] "RemoveContainer" containerID="ab2f636084ca233f0d2533ebbdef39359b533b4d9d6e57f3990093c9a81eee6b" Jul 10 22:52:17.744044 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:52:17.744022 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2f636084ca233f0d2533ebbdef39359b533b4d9d6e57f3990093c9a81eee6b\": container with ID starting with ab2f636084ca233f0d2533ebbdef39359b533b4d9d6e57f3990093c9a81eee6b not found: ID does not exist" containerID="ab2f636084ca233f0d2533ebbdef39359b533b4d9d6e57f3990093c9a81eee6b" Jul 10 22:52:17.744110 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.744054 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2f636084ca233f0d2533ebbdef39359b533b4d9d6e57f3990093c9a81eee6b"} err="failed to get container status \"ab2f636084ca233f0d2533ebbdef39359b533b4d9d6e57f3990093c9a81eee6b\": rpc error: code = NotFound desc = could not find container \"ab2f636084ca233f0d2533ebbdef39359b533b4d9d6e57f3990093c9a81eee6b\": container with ID starting with ab2f636084ca233f0d2533ebbdef39359b533b4d9d6e57f3990093c9a81eee6b not found: ID does not exist" Jul 10 22:52:17.747492 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.747446 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-workload-socket\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.747492 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.747473 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-token\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.747626 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.747495 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-podinfo\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.747626 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.747511 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c86pm\" (UniqueName: \"kubernetes.io/projected/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-kube-api-access-c86pm\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.747626 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.747599 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-workload-certs\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.747778 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.747658 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-credential-socket\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.747778 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.747689 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istiod-ca-cert\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.747778 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.747728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-data\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.747901 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.747821 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-envoy\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.750853 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.750827 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb"] Jul 10 22:52:17.762992 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.762968 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2ace46a5-openshift-default-7575f859c7-q72zb"] Jul 10 22:52:17.778047 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.778023 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:17.848581 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.848546 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-credential-socket\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.848581 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.848586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istiod-ca-cert\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.848877 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.848618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-data\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.848877 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.848734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-envoy\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.848877 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.848870 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-workload-socket\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.849086 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.848899 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-token\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.849086 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.848963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-podinfo\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.849086 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.849041 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-data\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.849225 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.849099 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-credential-socket\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.849225 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.848985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c86pm\" (UniqueName: \"kubernetes.io/projected/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-kube-api-access-c86pm\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.849328 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.849219 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-workload-socket\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.849457 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.849434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-workload-certs\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.849570 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.849553 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-workload-certs\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.849624 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.849614 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istiod-ca-cert\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.851067 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.851048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-envoy\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.851408 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.851388 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-podinfo\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.856720 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.856698 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-token\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.856970 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.856951 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c86pm\" (UniqueName: \"kubernetes.io/projected/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-kube-api-access-c86pm\") pod \"e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:17.989386 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:17.989335 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:18.133372 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:18.133344 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t"] Jul 10 22:52:18.134003 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:52:18.133968 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52ddfb3_b1e1_45f5_a7c6_9a57a5e08b7e.slice/crio-1c4497c6f7c7c3fd8eb89ff52c51d6ea8a70e80216bdefc6263ab9736e9b3aae WatchSource:0}: Error finding container 1c4497c6f7c7c3fd8eb89ff52c51d6ea8a70e80216bdefc6263ab9736e9b3aae: Status 404 returned error can't find the container with id 1c4497c6f7c7c3fd8eb89ff52c51d6ea8a70e80216bdefc6263ab9736e9b3aae Jul 10 22:52:18.136086 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:18.136045 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:52:18.136165 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:18.136126 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:52:18.136205 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:18.136172 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:52:18.365658 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:18.365494 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:52:18.385460 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:18.369860 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:52:18.740772 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:18.740734 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" event={"ID":"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e","Type":"ContainerStarted","Data":"edeed57a07d9ab4e44c7855deb06342b289df8b2c6b9e406169cadc968c7c6f2"} Jul 10 22:52:18.740961 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:18.740811 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" event={"ID":"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e","Type":"ContainerStarted","Data":"1c4497c6f7c7c3fd8eb89ff52c51d6ea8a70e80216bdefc6263ab9736e9b3aae"} Jul 10 22:52:18.760863 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:18.760814 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" podStartSLOduration=1.760783532 podStartE2EDuration="1.760783532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:52:18.759576143 +0000 UTC m=+1802.592460531" watchObservedRunningTime="2026-07-10 22:52:18.760783532 +0000 UTC m=+1802.593667914" Jul 10 22:52:18.878721 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:18.878686 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:18.944020 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:18.943974 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:18.990462 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:18.990428 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:19.311941 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.311910 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb"] Jul 10 22:52:19.316333 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.316309 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:19.318594 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.318573 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f617a605-serving-cert\"" Jul 10 22:52:19.326914 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.326889 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb"] Jul 10 22:52:19.363919 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.363889 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltvgx\" (UniqueName: \"kubernetes.io/projected/ff7916ec-2dbc-40f8-972b-074d49c9e613-kube-api-access-ltvgx\") pod \"maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb\" (UID: \"ff7916ec-2dbc-40f8-972b-074d49c9e613\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:19.364080 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.363932 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff7916ec-2dbc-40f8-972b-074d49c9e613-maas-api-tls\") pod \"maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb\" (UID: \"ff7916ec-2dbc-40f8-972b-074d49c9e613\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:19.364080 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.363950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7916ec-2dbc-40f8-972b-074d49c9e613-openshift-service-ca\") pod \"maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb\" (UID: \"ff7916ec-2dbc-40f8-972b-074d49c9e613\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:19.387004 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.386980 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:19.464429 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.464388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltvgx\" (UniqueName: \"kubernetes.io/projected/ff7916ec-2dbc-40f8-972b-074d49c9e613-kube-api-access-ltvgx\") pod \"maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb\" (UID: \"ff7916ec-2dbc-40f8-972b-074d49c9e613\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:19.464635 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.464436 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff7916ec-2dbc-40f8-972b-074d49c9e613-maas-api-tls\") pod \"maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb\" (UID: \"ff7916ec-2dbc-40f8-972b-074d49c9e613\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:19.464635 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.464458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7916ec-2dbc-40f8-972b-074d49c9e613-openshift-service-ca\") pod \"maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb\" (UID: \"ff7916ec-2dbc-40f8-972b-074d49c9e613\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:19.464635 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:52:19.464554 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-serving-cert: secret "maas-api-e2e-ait-f617a605-serving-cert" not found Jul 10 22:52:19.464635 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:52:19.464626 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff7916ec-2dbc-40f8-972b-074d49c9e613-maas-api-tls podName:ff7916ec-2dbc-40f8-972b-074d49c9e613 nodeName:}" failed. No retries permitted until 2026-07-10 22:52:19.964606444 +0000 UTC m=+1803.797490805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff7916ec-2dbc-40f8-972b-074d49c9e613-maas-api-tls") pod "maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" (UID: "ff7916ec-2dbc-40f8-972b-074d49c9e613") : secret "maas-api-e2e-ait-f617a605-serving-cert" not found Jul 10 22:52:19.465093 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.465075 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7916ec-2dbc-40f8-972b-074d49c9e613-openshift-service-ca\") pod \"maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb\" (UID: \"ff7916ec-2dbc-40f8-972b-074d49c9e613\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:19.475205 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.475184 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltvgx\" (UniqueName: \"kubernetes.io/projected/ff7916ec-2dbc-40f8-972b-074d49c9e613-kube-api-access-ltvgx\") pod \"maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb\" (UID: \"ff7916ec-2dbc-40f8-972b-074d49c9e613\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:19.622778 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.622699 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:19.690932 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.690898 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caaa290f-bdc0-4f3e-a2af-d99890e497b2" path="/var/lib/kubelet/pods/caaa290f-bdc0-4f3e-a2af-d99890e497b2/volumes" Jul 10 22:52:19.772482 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.772450 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:19.873111 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.873019 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:19.969542 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.969503 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff7916ec-2dbc-40f8-972b-074d49c9e613-maas-api-tls\") pod \"maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb\" (UID: \"ff7916ec-2dbc-40f8-972b-074d49c9e613\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:19.971993 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.971968 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff7916ec-2dbc-40f8-972b-074d49c9e613-maas-api-tls\") pod \"maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb\" (UID: \"ff7916ec-2dbc-40f8-972b-074d49c9e613\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:19.995136 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:19.995106 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:20.227953 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:20.227916 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:20.360957 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:20.360932 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb"] Jul 10 22:52:20.362530 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:52:20.362503 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7916ec_2dbc_40f8_972b_074d49c9e613.slice/crio-3552298b6fff973a3ce395523e542ca175a4ae19df3eb44c20561f7680d915ba WatchSource:0}: Error finding container 3552298b6fff973a3ce395523e542ca175a4ae19df3eb44c20561f7680d915ba: Status 404 returned error can't find the container with id 3552298b6fff973a3ce395523e542ca175a4ae19df3eb44c20561f7680d915ba Jul 10 22:52:20.363978 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:20.363960 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 22:52:20.752333 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:20.752297 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" event={"ID":"ff7916ec-2dbc-40f8-972b-074d49c9e613","Type":"ContainerStarted","Data":"3552298b6fff973a3ce395523e542ca175a4ae19df3eb44c20561f7680d915ba"} Jul 10 22:52:20.752546 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:20.752484 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:20.753446 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:20.753426 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:20.969524 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:20.969493 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:21.758712 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:21.758673 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" event={"ID":"ff7916ec-2dbc-40f8-972b-074d49c9e613","Type":"ContainerStarted","Data":"89fe583594b4ffc58acd45ef5707895ec09c05eb041a05954821059394b21fcd"} Jul 10 22:52:21.759195 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:21.758900 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:21.775860 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:21.775777 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" podStartSLOduration=2.2858658099999998 podStartE2EDuration="2.775764032s" totalImagesPullingTime="489.898222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:52:21.77393585 +0000 UTC m=+1805.606820249" watchObservedRunningTime="2026-07-10 22:52:21.775764032 +0000 UTC m=+1805.608648415" Jul 10 22:52:23.855510 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:23.855477 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb"] Jul 10 22:52:23.855989 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:23.855715 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" podUID="ff7916ec-2dbc-40f8-972b-074d49c9e613" containerName="maas-api" containerID="cri-o://89fe583594b4ffc58acd45ef5707895ec09c05eb041a05954821059394b21fcd" gracePeriod=30 Jul 10 22:52:23.876897 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:23.876863 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:23.926187 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:23.926153 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:24.105755 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.105695 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:24.206463 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.206428 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltvgx\" (UniqueName: \"kubernetes.io/projected/ff7916ec-2dbc-40f8-972b-074d49c9e613-kube-api-access-ltvgx\") pod \"ff7916ec-2dbc-40f8-972b-074d49c9e613\" (UID: \"ff7916ec-2dbc-40f8-972b-074d49c9e613\") " Jul 10 22:52:24.206666 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.206512 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7916ec-2dbc-40f8-972b-074d49c9e613-openshift-service-ca\") pod \"ff7916ec-2dbc-40f8-972b-074d49c9e613\" (UID: \"ff7916ec-2dbc-40f8-972b-074d49c9e613\") " Jul 10 22:52:24.206666 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.206540 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff7916ec-2dbc-40f8-972b-074d49c9e613-maas-api-tls\") pod \"ff7916ec-2dbc-40f8-972b-074d49c9e613\" (UID: \"ff7916ec-2dbc-40f8-972b-074d49c9e613\") " Jul 10 22:52:24.206918 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.206894 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff7916ec-2dbc-40f8-972b-074d49c9e613-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff7916ec-2dbc-40f8-972b-074d49c9e613" (UID: "ff7916ec-2dbc-40f8-972b-074d49c9e613"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:52:24.208733 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.208703 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff7916ec-2dbc-40f8-972b-074d49c9e613-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff7916ec-2dbc-40f8-972b-074d49c9e613" (UID: "ff7916ec-2dbc-40f8-972b-074d49c9e613"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:52:24.208851 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.208770 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7916ec-2dbc-40f8-972b-074d49c9e613-kube-api-access-ltvgx" (OuterVolumeSpecName: "kube-api-access-ltvgx") pod "ff7916ec-2dbc-40f8-972b-074d49c9e613" (UID: "ff7916ec-2dbc-40f8-972b-074d49c9e613"). InnerVolumeSpecName "kube-api-access-ltvgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:52:24.307147 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.307107 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltvgx\" (UniqueName: \"kubernetes.io/projected/ff7916ec-2dbc-40f8-972b-074d49c9e613-kube-api-access-ltvgx\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:24.307147 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.307139 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7916ec-2dbc-40f8-972b-074d49c9e613-openshift-service-ca\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:24.307147 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.307149 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff7916ec-2dbc-40f8-972b-074d49c9e613-maas-api-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:24.775359 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.775317 2565 generic.go:356] "Generic (PLEG): container finished" podID="ff7916ec-2dbc-40f8-972b-074d49c9e613" containerID="89fe583594b4ffc58acd45ef5707895ec09c05eb041a05954821059394b21fcd" exitCode=0 Jul 10 22:52:24.775563 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.775367 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" Jul 10 22:52:24.775563 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.775398 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" event={"ID":"ff7916ec-2dbc-40f8-972b-074d49c9e613","Type":"ContainerDied","Data":"89fe583594b4ffc58acd45ef5707895ec09c05eb041a05954821059394b21fcd"} Jul 10 22:52:24.775563 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.775448 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb" event={"ID":"ff7916ec-2dbc-40f8-972b-074d49c9e613","Type":"ContainerDied","Data":"3552298b6fff973a3ce395523e542ca175a4ae19df3eb44c20561f7680d915ba"} Jul 10 22:52:24.775563 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.775471 2565 scope.go:117] "RemoveContainer" containerID="89fe583594b4ffc58acd45ef5707895ec09c05eb041a05954821059394b21fcd" Jul 10 22:52:24.785561 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.785544 2565 scope.go:117] "RemoveContainer" containerID="89fe583594b4ffc58acd45ef5707895ec09c05eb041a05954821059394b21fcd" Jul 10 22:52:24.785851 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:52:24.785825 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89fe583594b4ffc58acd45ef5707895ec09c05eb041a05954821059394b21fcd\": container with ID starting with 89fe583594b4ffc58acd45ef5707895ec09c05eb041a05954821059394b21fcd not found: ID does not exist" containerID="89fe583594b4ffc58acd45ef5707895ec09c05eb041a05954821059394b21fcd" Jul 10 22:52:24.785909 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.785850 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89fe583594b4ffc58acd45ef5707895ec09c05eb041a05954821059394b21fcd"} err="failed to get container status \"89fe583594b4ffc58acd45ef5707895ec09c05eb041a05954821059394b21fcd\": rpc error: code = NotFound desc = could not find container \"89fe583594b4ffc58acd45ef5707895ec09c05eb041a05954821059394b21fcd\": container with ID starting with 89fe583594b4ffc58acd45ef5707895ec09c05eb041a05954821059394b21fcd not found: ID does not exist" Jul 10 22:52:24.797228 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.797206 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb"] Jul 10 22:52:24.801052 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.801033 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f617a605-6c9968f4db-rdtpb"] Jul 10 22:52:24.830403 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.830374 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t"] Jul 10 22:52:24.830559 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.830539 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" podUID="f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" containerName="istio-proxy" containerID="cri-o://edeed57a07d9ab4e44c7855deb06342b289df8b2c6b9e406169cadc968c7c6f2" gracePeriod=30 Jul 10 22:52:24.851179 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.851149 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:24.949693 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:24.949656 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:25.691345 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:25.691312 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff7916ec-2dbc-40f8-972b-074d49c9e613" path="/var/lib/kubelet/pods/ff7916ec-2dbc-40f8-972b-074d49c9e613/volumes" Jul 10 22:52:26.270757 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:26.270727 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:30.087107 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.087076 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:30.161185 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161083 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-data\") pod \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " Jul 10 22:52:30.161185 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161128 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-token\") pod \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " Jul 10 22:52:30.161185 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161147 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-credential-socket\") pod \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " Jul 10 22:52:30.161185 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161187 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-workload-socket\") pod \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " Jul 10 22:52:30.161521 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161300 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-podinfo\") pod \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " Jul 10 22:52:30.161521 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161342 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-workload-certs\") pod \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " Jul 10 22:52:30.161521 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161365 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istiod-ca-cert\") pod \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " Jul 10 22:52:30.161521 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161398 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c86pm\" (UniqueName: \"kubernetes.io/projected/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-kube-api-access-c86pm\") pod \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " Jul 10 22:52:30.161521 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161424 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-data" (OuterVolumeSpecName: "istio-data") pod "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" (UID: "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:30.161521 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161475 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-envoy\") pod \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\" (UID: \"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e\") " Jul 10 22:52:30.161836 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161557 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" (UID: "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:30.161836 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161571 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" (UID: "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:30.161836 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161691 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" (UID: "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:30.161836 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161824 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-data\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:30.162050 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161850 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-credential-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:30.162050 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161865 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-workload-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:30.162050 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161879 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-workload-certs\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:30.162050 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.161823 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" (UID: "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:52:30.163553 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.163524 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-kube-api-access-c86pm" (OuterVolumeSpecName: "kube-api-access-c86pm") pod "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" (UID: "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e"). InnerVolumeSpecName "kube-api-access-c86pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:52:30.163664 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.163524 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-token" (OuterVolumeSpecName: "istio-token") pod "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" (UID: "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:52:30.163838 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.163818 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" (UID: "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 22:52:30.163898 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.163835 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" (UID: "f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:30.262529 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.262489 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istiod-ca-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:30.262529 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.262521 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c86pm\" (UniqueName: \"kubernetes.io/projected/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-kube-api-access-c86pm\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:30.262529 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.262534 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-envoy\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:30.262771 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.262545 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-token\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:30.262771 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.262557 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e-istio-podinfo\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:30.605214 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.605177 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:30.666952 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.666913 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:30.760145 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.760107 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k"] Jul 10 22:52:30.760712 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.760682 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" containerName="istio-proxy" Jul 10 22:52:30.760712 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.760712 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" containerName="istio-proxy" Jul 10 22:52:30.760948 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.760758 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff7916ec-2dbc-40f8-972b-074d49c9e613" containerName="maas-api" Jul 10 22:52:30.760948 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.760767 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff7916ec-2dbc-40f8-972b-074d49c9e613" containerName="maas-api" Jul 10 22:52:30.760948 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.760882 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff7916ec-2dbc-40f8-972b-074d49c9e613" containerName="maas-api" Jul 10 22:52:30.760948 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.760896 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" containerName="istio-proxy" Jul 10 22:52:30.764549 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.764517 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.767072 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.767051 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-de4fec61-openshift-default-dockercfg-6frvm\"" Jul 10 22:52:30.773989 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.773964 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k"] Jul 10 22:52:30.803783 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.803716 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:30.812441 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.812415 2565 generic.go:356] "Generic (PLEG): container finished" podID="f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" containerID="edeed57a07d9ab4e44c7855deb06342b289df8b2c6b9e406169cadc968c7c6f2" exitCode=0 Jul 10 22:52:30.812573 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.812457 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" Jul 10 22:52:30.812573 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.812502 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" event={"ID":"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e","Type":"ContainerDied","Data":"edeed57a07d9ab4e44c7855deb06342b289df8b2c6b9e406169cadc968c7c6f2"} Jul 10 22:52:30.812573 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.812565 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t" event={"ID":"f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e","Type":"ContainerDied","Data":"1c4497c6f7c7c3fd8eb89ff52c51d6ea8a70e80216bdefc6263ab9736e9b3aae"} Jul 10 22:52:30.812750 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.812599 2565 scope.go:117] "RemoveContainer" containerID="edeed57a07d9ab4e44c7855deb06342b289df8b2c6b9e406169cadc968c7c6f2" Jul 10 22:52:30.821666 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.821647 2565 scope.go:117] "RemoveContainer" containerID="edeed57a07d9ab4e44c7855deb06342b289df8b2c6b9e406169cadc968c7c6f2" Jul 10 22:52:30.821979 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:52:30.821960 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edeed57a07d9ab4e44c7855deb06342b289df8b2c6b9e406169cadc968c7c6f2\": container with ID starting with edeed57a07d9ab4e44c7855deb06342b289df8b2c6b9e406169cadc968c7c6f2 not found: ID does not exist" containerID="edeed57a07d9ab4e44c7855deb06342b289df8b2c6b9e406169cadc968c7c6f2" Jul 10 22:52:30.822053 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.821990 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edeed57a07d9ab4e44c7855deb06342b289df8b2c6b9e406169cadc968c7c6f2"} err="failed to get container status \"edeed57a07d9ab4e44c7855deb06342b289df8b2c6b9e406169cadc968c7c6f2\": rpc error: code = NotFound desc = could not find container \"edeed57a07d9ab4e44c7855deb06342b289df8b2c6b9e406169cadc968c7c6f2\": container with ID starting with edeed57a07d9ab4e44c7855deb06342b289df8b2c6b9e406169cadc968c7c6f2 not found: ID does not exist" Jul 10 22:52:30.835615 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.835591 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t"] Jul 10 22:52:30.839514 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.839491 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f617a605-openshift-default-54d9b8cbdc-46h8t"] Jul 10 22:52:30.859575 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.859503 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:30.867435 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.867409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-workload-certs\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.867627 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.867463 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-token\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.867627 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.867529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-podinfo\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.867627 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.867575 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-envoy\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.867627 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.867616 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-credential-socket\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.868072 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.867708 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-data\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.868072 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.867764 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-workload-socket\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.868072 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.867830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjpcz\" (UniqueName: \"kubernetes.io/projected/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-kube-api-access-cjpcz\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.868072 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.867861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istiod-ca-cert\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.968931 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.968884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-token\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.968931 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.968940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-podinfo\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.969169 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.968961 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-envoy\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.969169 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.968983 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-credential-socket\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.969169 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.969137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-data\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.969403 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.969170 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-workload-socket\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.969403 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.969199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjpcz\" (UniqueName: \"kubernetes.io/projected/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-kube-api-access-cjpcz\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.969403 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.969246 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istiod-ca-cert\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.969403 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.969294 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-workload-certs\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.969403 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.969390 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-credential-socket\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.969660 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.969519 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-data\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.969660 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.969568 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-workload-socket\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.969660 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.969598 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-workload-certs\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.969998 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.969977 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istiod-ca-cert\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.971326 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.971305 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-envoy\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.971546 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.971529 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-podinfo\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.976314 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.976288 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-token\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:30.976611 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:30.976595 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjpcz\" (UniqueName: \"kubernetes.io/projected/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-kube-api-access-cjpcz\") pod \"e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:31.078896 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:31.078864 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:31.214112 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:31.214082 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k"] Jul 10 22:52:31.215517 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:52:31.215485 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1666c9_ad35_4057_8f99_2ffb8f0ae628.slice/crio-5b808951b8535363fc87c10c1799f467c0bef60ba68fdd736ed89faeb3782a5a WatchSource:0}: Error finding container 5b808951b8535363fc87c10c1799f467c0bef60ba68fdd736ed89faeb3782a5a: Status 404 returned error can't find the container with id 5b808951b8535363fc87c10c1799f467c0bef60ba68fdd736ed89faeb3782a5a Jul 10 22:52:31.217850 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:31.217777 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:52:31.217945 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:31.217899 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:52:31.218016 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:31.217955 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:52:31.692529 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:31.692496 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e" path="/var/lib/kubelet/pods/f52ddfb3-b1e1-45f5-a7c6-9a57a5e08b7e/volumes" Jul 10 22:52:31.819698 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:31.819653 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" event={"ID":"7c1666c9-ad35-4057-8f99-2ffb8f0ae628","Type":"ContainerStarted","Data":"219151ebc468f15959457c9d151761f8f8ea298c1fdec63c7a5bf5daff3931fb"} Jul 10 22:52:31.819909 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:31.819726 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" event={"ID":"7c1666c9-ad35-4057-8f99-2ffb8f0ae628","Type":"ContainerStarted","Data":"5b808951b8535363fc87c10c1799f467c0bef60ba68fdd736ed89faeb3782a5a"} Jul 10 22:52:31.841282 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:31.841217 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" podStartSLOduration=1.8411995060000002 podStartE2EDuration="1.841199506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:52:31.838766866 +0000 UTC m=+1815.671651250" watchObservedRunningTime="2026-07-10 22:52:31.841199506 +0000 UTC m=+1815.674083958" Jul 10 22:52:31.899956 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:31.899922 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:31.999151 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:31.999104 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:32.079487 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:32.079456 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:32.084625 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:32.084598 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:32.825805 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:32.825774 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:32.826647 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:32.826631 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:33.850054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:33.850000 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:36.341423 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:36.341379 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:36.506335 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:36.506300 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k"] Jul 10 22:52:36.506524 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:36.506499 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" podUID="7c1666c9-ad35-4057-8f99-2ffb8f0ae628" containerName="istio-proxy" containerID="cri-o://219151ebc468f15959457c9d151761f8f8ea298c1fdec63c7a5bf5daff3931fb" gracePeriod=30 Jul 10 22:52:36.525948 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:36.525905 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:36.630930 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:36.630839 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:38.064655 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:38.064566 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:41.749228 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.749199 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:41.866518 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.866428 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-workload-certs\") pod \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " Jul 10 22:52:41.866518 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.866475 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-data\") pod \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " Jul 10 22:52:41.866518 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.866508 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-workload-socket\") pod \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " Jul 10 22:52:41.866856 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.866569 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-envoy\") pod \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " Jul 10 22:52:41.866856 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.866594 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istiod-ca-cert\") pod \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " Jul 10 22:52:41.866856 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.866626 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-token\") pod \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " Jul 10 22:52:41.866856 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.866740 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-podinfo\") pod \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " Jul 10 22:52:41.866856 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.866768 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-credential-socket\") pod \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " Jul 10 22:52:41.866856 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.866787 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c1666c9-ad35-4057-8f99-2ffb8f0ae628" (UID: "7c1666c9-ad35-4057-8f99-2ffb8f0ae628"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:41.866856 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.866828 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c1666c9-ad35-4057-8f99-2ffb8f0ae628" (UID: "7c1666c9-ad35-4057-8f99-2ffb8f0ae628"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:41.867208 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.866822 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjpcz\" (UniqueName: \"kubernetes.io/projected/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-kube-api-access-cjpcz\") pod \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\" (UID: \"7c1666c9-ad35-4057-8f99-2ffb8f0ae628\") " Jul 10 22:52:41.867208 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.866899 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c1666c9-ad35-4057-8f99-2ffb8f0ae628" (UID: "7c1666c9-ad35-4057-8f99-2ffb8f0ae628"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:41.867208 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.867182 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-workload-certs\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:41.867208 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.867206 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-data\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:41.867426 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.867219 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-workload-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:41.867426 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.867374 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c1666c9-ad35-4057-8f99-2ffb8f0ae628" (UID: "7c1666c9-ad35-4057-8f99-2ffb8f0ae628"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:41.867538 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.867490 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c1666c9-ad35-4057-8f99-2ffb8f0ae628" (UID: "7c1666c9-ad35-4057-8f99-2ffb8f0ae628"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:52:41.869450 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.869398 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c1666c9-ad35-4057-8f99-2ffb8f0ae628" (UID: "7c1666c9-ad35-4057-8f99-2ffb8f0ae628"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 22:52:41.869579 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.869508 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c1666c9-ad35-4057-8f99-2ffb8f0ae628" (UID: "7c1666c9-ad35-4057-8f99-2ffb8f0ae628"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:52:41.869579 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.869550 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c1666c9-ad35-4057-8f99-2ffb8f0ae628" (UID: "7c1666c9-ad35-4057-8f99-2ffb8f0ae628"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:52:41.869701 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.869662 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-kube-api-access-cjpcz" (OuterVolumeSpecName: "kube-api-access-cjpcz") pod "7c1666c9-ad35-4057-8f99-2ffb8f0ae628" (UID: "7c1666c9-ad35-4057-8f99-2ffb8f0ae628"). InnerVolumeSpecName "kube-api-access-cjpcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:52:41.873734 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.873700 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c1666c9-ad35-4057-8f99-2ffb8f0ae628" containerID="219151ebc468f15959457c9d151761f8f8ea298c1fdec63c7a5bf5daff3931fb" exitCode=0 Jul 10 22:52:41.873734 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.873719 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" Jul 10 22:52:41.873952 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.873767 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" event={"ID":"7c1666c9-ad35-4057-8f99-2ffb8f0ae628","Type":"ContainerDied","Data":"219151ebc468f15959457c9d151761f8f8ea298c1fdec63c7a5bf5daff3931fb"} Jul 10 22:52:41.873952 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.873862 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k" event={"ID":"7c1666c9-ad35-4057-8f99-2ffb8f0ae628","Type":"ContainerDied","Data":"5b808951b8535363fc87c10c1799f467c0bef60ba68fdd736ed89faeb3782a5a"} Jul 10 22:52:41.873952 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.873892 2565 scope.go:117] "RemoveContainer" containerID="219151ebc468f15959457c9d151761f8f8ea298c1fdec63c7a5bf5daff3931fb" Jul 10 22:52:41.891987 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.891965 2565 scope.go:117] "RemoveContainer" containerID="219151ebc468f15959457c9d151761f8f8ea298c1fdec63c7a5bf5daff3931fb" Jul 10 22:52:41.892333 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:52:41.892312 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219151ebc468f15959457c9d151761f8f8ea298c1fdec63c7a5bf5daff3931fb\": container with ID starting with 219151ebc468f15959457c9d151761f8f8ea298c1fdec63c7a5bf5daff3931fb not found: ID does not exist" containerID="219151ebc468f15959457c9d151761f8f8ea298c1fdec63c7a5bf5daff3931fb" Jul 10 22:52:41.892388 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.892344 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219151ebc468f15959457c9d151761f8f8ea298c1fdec63c7a5bf5daff3931fb"} err="failed to get container status \"219151ebc468f15959457c9d151761f8f8ea298c1fdec63c7a5bf5daff3931fb\": rpc error: code = NotFound desc = could not find container \"219151ebc468f15959457c9d151761f8f8ea298c1fdec63c7a5bf5daff3931fb\": container with ID starting with 219151ebc468f15959457c9d151761f8f8ea298c1fdec63c7a5bf5daff3931fb not found: ID does not exist" Jul 10 22:52:41.905459 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.903824 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k"] Jul 10 22:52:41.909821 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.909780 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-de4fec61-openshift-default-6fc7764b4c-qkh4k"] Jul 10 22:52:41.967809 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.967766 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-podinfo\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:41.967809 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.967809 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-credential-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:41.967809 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.967820 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjpcz\" (UniqueName: \"kubernetes.io/projected/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-kube-api-access-cjpcz\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:41.968044 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.967830 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-envoy\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:41.968044 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.967839 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istiod-ca-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:41.968044 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:41.967847 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1666c9-ad35-4057-8f99-2ffb8f0ae628-istio-token\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:52:42.386256 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:42.386216 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:42.486831 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:42.486788 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:42.863004 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:42.862970 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:42.963065 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:42.963023 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:43.062344 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:43.062305 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:43.690810 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:43.690770 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c1666c9-ad35-4057-8f99-2ffb8f0ae628" path="/var/lib/kubelet/pods/7c1666c9-ad35-4057-8f99-2ffb8f0ae628/volumes" Jul 10 22:52:44.456784 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:44.456748 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:50.057581 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:50.057546 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:50.157946 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:50.157913 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:50.257450 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:50.257420 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:50.358301 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:50.358215 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:52:50.725325 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:52:50.725293 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:53:16.715517 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:53:16.715484 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:53:17.517684 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:53:17.517651 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:53:22.366885 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:53:22.366851 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:53:22.538017 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:53:22.537981 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:53:22.663653 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:53:22.663562 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:53:38.487290 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:53:38.487200 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:53:44.299891 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:53:44.299852 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:53:44.358087 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:53:44.358055 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:53:44.427919 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:53:44.427885 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:53:44.526111 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:53:44.526071 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:53:55.009065 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:53:55.009029 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:53:55.793778 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:53:55.793742 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:01.478688 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:01.478634 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:01.594176 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:01.594108 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:01.686853 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:01.686780 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:01.782874 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:01.782772 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:02.522955 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:02.522919 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:02.634214 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:02.634181 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:02.778189 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:02.778107 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:04.341622 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:04.341588 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:10.173539 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:10.173504 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:10.275333 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:10.275297 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:10.844085 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:10.844050 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:16.763042 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:16.763008 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:16.865069 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:16.865037 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:16.990008 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:16.989974 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:17.284058 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.284022 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs"] Jul 10 22:54:17.284393 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.284381 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c1666c9-ad35-4057-8f99-2ffb8f0ae628" containerName="istio-proxy" Jul 10 22:54:17.284438 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.284394 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c1666c9-ad35-4057-8f99-2ffb8f0ae628" containerName="istio-proxy" Jul 10 22:54:17.284494 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.284485 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c1666c9-ad35-4057-8f99-2ffb8f0ae628" containerName="istio-proxy" Jul 10 22:54:17.287557 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.287540 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.289869 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.289846 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-26b61da1-openshift-default-dockercfg-rh2d8\"" Jul 10 22:54:17.298262 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.298229 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:17.301390 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.301365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-envoy\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.301471 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.301410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-workload-socket\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.301471 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.301429 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istiod-ca-cert\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.301540 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.301474 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-podinfo\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.301540 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.301490 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjgf7\" (UniqueName: \"kubernetes.io/projected/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-kube-api-access-qjgf7\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.301607 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.301551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-workload-certs\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.301607 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.301572 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-data\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.301671 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.301605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-token\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.301671 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.301622 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-credential-socket\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.311865 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.311842 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs"] Jul 10 22:54:17.402060 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402024 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-workload-socket\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.402060 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istiod-ca-cert\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.402303 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-podinfo\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.402303 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjgf7\" (UniqueName: \"kubernetes.io/projected/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-kube-api-access-qjgf7\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.402303 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-workload-certs\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.402303 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-data\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.402303 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402219 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-token\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.402557 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-credential-socket\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.402557 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402514 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-workload-socket\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.402557 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402545 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-data\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.402720 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402521 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-envoy\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.402720 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402595 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-workload-certs\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.402851 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402773 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-credential-socket\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.402907 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.402894 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istiod-ca-cert\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.404500 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.404481 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-envoy\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.404642 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.404623 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-podinfo\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.413091 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.413065 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjgf7\" (UniqueName: \"kubernetes.io/projected/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-kube-api-access-qjgf7\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.413223 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.413151 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-token\") pod \"e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.599680 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.599607 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-26b61da1-openshift-default-dockercfg-rh2d8\"" Jul 10 22:54:17.607928 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.607901 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:17.749740 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.749715 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs"] Jul 10 22:54:17.751319 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:54:17.751294 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fcefb4_13d3_4e1a_aae6_9ca586d9cc0b.slice/crio-5db77a40fe6c47fd2ffcaa381fdd7a680bcee52c85fb4636661cdc2fdc64d1fe WatchSource:0}: Error finding container 5db77a40fe6c47fd2ffcaa381fdd7a680bcee52c85fb4636661cdc2fdc64d1fe: Status 404 returned error can't find the container with id 5db77a40fe6c47fd2ffcaa381fdd7a680bcee52c85fb4636661cdc2fdc64d1fe Jul 10 22:54:17.753417 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.753383 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:54:17.753472 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.753451 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:54:17.753515 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.753481 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:54:17.930264 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:17.930169 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:18.357011 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:18.356968 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" event={"ID":"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b","Type":"ContainerStarted","Data":"6441ed391fbd74bf98c617bc0cb92678dfdf5e67f889b06b5d5c65ad0f62af67"} Jul 10 22:54:18.357174 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:18.357051 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" event={"ID":"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b","Type":"ContainerStarted","Data":"5db77a40fe6c47fd2ffcaa381fdd7a680bcee52c85fb4636661cdc2fdc64d1fe"} Jul 10 22:54:18.380859 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:18.380787 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" podStartSLOduration=1.380772986 podStartE2EDuration="1.380772986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:54:18.378229041 +0000 UTC m=+1922.211113424" watchObservedRunningTime="2026-07-10 22:54:18.380772986 +0000 UTC m=+1922.213657369" Jul 10 22:54:18.608717 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:18.608630 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:18.613592 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:18.613568 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:19.362652 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:19.362626 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:19.363846 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:19.363828 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:20.264199 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.264154 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:20.364750 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.364711 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:20.443189 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.443159 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb"] Jul 10 22:54:20.446443 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.446427 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:20.448616 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.448593 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-26b61da1-serving-cert\"" Jul 10 22:54:20.455910 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.455885 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb"] Jul 10 22:54:20.477082 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.477044 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:20.530318 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.530227 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bc7fab5-7f5c-489c-a6da-77b459ddd036-maas-api-tls\") pod \"maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb\" (UID: \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:20.530318 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.530270 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc7fab5-7f5c-489c-a6da-77b459ddd036-openshift-service-ca\") pod \"maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb\" (UID: \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:20.530502 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.530334 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp2cg\" (UniqueName: \"kubernetes.io/projected/3bc7fab5-7f5c-489c-a6da-77b459ddd036-kube-api-access-bp2cg\") pod \"maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb\" (UID: \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:20.631785 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.631743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bc7fab5-7f5c-489c-a6da-77b459ddd036-maas-api-tls\") pod \"maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb\" (UID: \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:20.631990 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.631821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc7fab5-7f5c-489c-a6da-77b459ddd036-openshift-service-ca\") pod \"maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb\" (UID: \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:20.631990 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.631868 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp2cg\" (UniqueName: \"kubernetes.io/projected/3bc7fab5-7f5c-489c-a6da-77b459ddd036-kube-api-access-bp2cg\") pod \"maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb\" (UID: \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:20.631990 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:54:20.631903 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-serving-cert: secret "maas-api-e2e-mt-26b61da1-serving-cert" not found Jul 10 22:54:20.631990 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:54:20.631979 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bc7fab5-7f5c-489c-a6da-77b459ddd036-maas-api-tls podName:3bc7fab5-7f5c-489c-a6da-77b459ddd036 nodeName:}" failed. No retries permitted until 2026-07-10 22:54:21.131956749 +0000 UTC m=+1924.964841132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3bc7fab5-7f5c-489c-a6da-77b459ddd036-maas-api-tls") pod "maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" (UID: "3bc7fab5-7f5c-489c-a6da-77b459ddd036") : secret "maas-api-e2e-mt-26b61da1-serving-cert" not found Jul 10 22:54:20.632474 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.632455 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc7fab5-7f5c-489c-a6da-77b459ddd036-openshift-service-ca\") pod \"maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb\" (UID: \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:20.640534 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:20.640510 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp2cg\" (UniqueName: \"kubernetes.io/projected/3bc7fab5-7f5c-489c-a6da-77b459ddd036-kube-api-access-bp2cg\") pod \"maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb\" (UID: \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:21.036411 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:21.036375 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:21.120693 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:21.120655 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:21.136014 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:21.135980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bc7fab5-7f5c-489c-a6da-77b459ddd036-maas-api-tls\") pod \"maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb\" (UID: \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:21.138447 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:21.138426 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bc7fab5-7f5c-489c-a6da-77b459ddd036-maas-api-tls\") pod \"maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb\" (UID: \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:21.222406 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:21.222376 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:21.359076 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:21.359001 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:21.496013 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:21.495980 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb"] Jul 10 22:54:21.496840 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:54:21.496784 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc7fab5_7f5c_489c_a6da_77b459ddd036.slice/crio-d58a7ddfd6e173033af4c49432db0c9cc4b71dfd4f121a59770a24aad97418e9 WatchSource:0}: Error finding container d58a7ddfd6e173033af4c49432db0c9cc4b71dfd4f121a59770a24aad97418e9: Status 404 returned error can't find the container with id d58a7ddfd6e173033af4c49432db0c9cc4b71dfd4f121a59770a24aad97418e9 Jul 10 22:54:22.380955 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:22.380910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" event={"ID":"3bc7fab5-7f5c-489c-a6da-77b459ddd036","Type":"ContainerStarted","Data":"f0df30af184986384ff306bcde5312040d8f97c76a4def8d8ccd45c8e4911966"} Jul 10 22:54:22.381138 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:22.380978 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" event={"ID":"3bc7fab5-7f5c-489c-a6da-77b459ddd036","Type":"ContainerStarted","Data":"d58a7ddfd6e173033af4c49432db0c9cc4b71dfd4f121a59770a24aad97418e9"} Jul 10 22:54:22.381138 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:22.381124 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:22.400168 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:22.400115 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" podStartSLOduration=1.9786272280000001 podStartE2EDuration="2.400099949s" totalImagesPullingTime="421.472721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:54:22.397309542 +0000 UTC m=+1926.230193948" watchObservedRunningTime="2026-07-10 22:54:22.400099949 +0000 UTC m=+1926.232984331" Jul 10 22:54:24.757592 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:24.757561 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:25.418140 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:25.418103 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:25.517304 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:25.517273 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:25.841859 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:25.841824 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:28.391047 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:28.391010 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:31.438226 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:31.438182 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:31.494749 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:31.494708 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:31.879858 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:31.879822 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb"] Jul 10 22:54:31.880092 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:31.880065 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" podUID="3bc7fab5-7f5c-489c-a6da-77b459ddd036" containerName="maas-api" containerID="cri-o://f0df30af184986384ff306bcde5312040d8f97c76a4def8d8ccd45c8e4911966" gracePeriod=30 Jul 10 22:54:31.915434 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:31.915395 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:31.990714 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:31.990636 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:32.133805 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.133762 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:32.235313 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.235273 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc7fab5-7f5c-489c-a6da-77b459ddd036-openshift-service-ca\") pod \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\" (UID: \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\") " Jul 10 22:54:32.235491 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.235340 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bc7fab5-7f5c-489c-a6da-77b459ddd036-maas-api-tls\") pod \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\" (UID: \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\") " Jul 10 22:54:32.235491 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.235390 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp2cg\" (UniqueName: \"kubernetes.io/projected/3bc7fab5-7f5c-489c-a6da-77b459ddd036-kube-api-access-bp2cg\") pod \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\" (UID: \"3bc7fab5-7f5c-489c-a6da-77b459ddd036\") " Jul 10 22:54:32.235835 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.235772 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc7fab5-7f5c-489c-a6da-77b459ddd036-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3bc7fab5-7f5c-489c-a6da-77b459ddd036" (UID: "3bc7fab5-7f5c-489c-a6da-77b459ddd036"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:54:32.237485 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.237456 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc7fab5-7f5c-489c-a6da-77b459ddd036-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3bc7fab5-7f5c-489c-a6da-77b459ddd036" (UID: "3bc7fab5-7f5c-489c-a6da-77b459ddd036"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:54:32.237485 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.237464 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc7fab5-7f5c-489c-a6da-77b459ddd036-kube-api-access-bp2cg" (OuterVolumeSpecName: "kube-api-access-bp2cg") pod "3bc7fab5-7f5c-489c-a6da-77b459ddd036" (UID: "3bc7fab5-7f5c-489c-a6da-77b459ddd036"). InnerVolumeSpecName "kube-api-access-bp2cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:54:32.330063 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.330013 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:32.336215 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.336184 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc7fab5-7f5c-489c-a6da-77b459ddd036-openshift-service-ca\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:54:32.336215 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.336216 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bc7fab5-7f5c-489c-a6da-77b459ddd036-maas-api-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:54:32.336419 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.336229 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp2cg\" (UniqueName: \"kubernetes.io/projected/3bc7fab5-7f5c-489c-a6da-77b459ddd036-kube-api-access-bp2cg\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:54:32.443341 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.443303 2565 generic.go:356] "Generic (PLEG): container finished" podID="3bc7fab5-7f5c-489c-a6da-77b459ddd036" containerID="f0df30af184986384ff306bcde5312040d8f97c76a4def8d8ccd45c8e4911966" exitCode=0 Jul 10 22:54:32.443830 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.443378 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" Jul 10 22:54:32.443830 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.443383 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" event={"ID":"3bc7fab5-7f5c-489c-a6da-77b459ddd036","Type":"ContainerDied","Data":"f0df30af184986384ff306bcde5312040d8f97c76a4def8d8ccd45c8e4911966"} Jul 10 22:54:32.443830 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.443484 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb" event={"ID":"3bc7fab5-7f5c-489c-a6da-77b459ddd036","Type":"ContainerDied","Data":"d58a7ddfd6e173033af4c49432db0c9cc4b71dfd4f121a59770a24aad97418e9"} Jul 10 22:54:32.443830 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.443519 2565 scope.go:117] "RemoveContainer" containerID="f0df30af184986384ff306bcde5312040d8f97c76a4def8d8ccd45c8e4911966" Jul 10 22:54:32.453183 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.453163 2565 scope.go:117] "RemoveContainer" containerID="f0df30af184986384ff306bcde5312040d8f97c76a4def8d8ccd45c8e4911966" Jul 10 22:54:32.453487 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:54:32.453458 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0df30af184986384ff306bcde5312040d8f97c76a4def8d8ccd45c8e4911966\": container with ID starting with f0df30af184986384ff306bcde5312040d8f97c76a4def8d8ccd45c8e4911966 not found: ID does not exist" containerID="f0df30af184986384ff306bcde5312040d8f97c76a4def8d8ccd45c8e4911966" Jul 10 22:54:32.453585 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.453497 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0df30af184986384ff306bcde5312040d8f97c76a4def8d8ccd45c8e4911966"} err="failed to get container status \"f0df30af184986384ff306bcde5312040d8f97c76a4def8d8ccd45c8e4911966\": rpc error: code = NotFound desc = could not find container \"f0df30af184986384ff306bcde5312040d8f97c76a4def8d8ccd45c8e4911966\": container with ID starting with f0df30af184986384ff306bcde5312040d8f97c76a4def8d8ccd45c8e4911966 not found: ID does not exist" Jul 10 22:54:32.467718 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.467693 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb"] Jul 10 22:54:32.469956 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:32.469937 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26b61da1-7cf794dc8b-4rgtb"] Jul 10 22:54:33.691054 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:33.691022 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bc7fab5-7f5c-489c-a6da-77b459ddd036" path="/var/lib/kubelet/pods/3bc7fab5-7f5c-489c-a6da-77b459ddd036/volumes" Jul 10 22:54:34.541783 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:34.541366 2565 patch_prober.go:28] interesting pod/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:39860->10.134.0.54:15021: read: connection reset by peer" start-of-body= Jul 10 22:54:34.541783 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:34.541434 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" podUID="f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:39860->10.134.0.54:15021: read: connection reset by peer" Jul 10 22:54:35.462098 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:35.462060 2565 generic.go:356] "Generic (PLEG): container finished" podID="f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" containerID="6441ed391fbd74bf98c617bc0cb92678dfdf5e67f889b06b5d5c65ad0f62af67" exitCode=0 Jul 10 22:54:35.462561 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:35.462152 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" event={"ID":"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b","Type":"ContainerDied","Data":"6441ed391fbd74bf98c617bc0cb92678dfdf5e67f889b06b5d5c65ad0f62af67"} Jul 10 22:54:35.462612 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:35.462564 2565 scope.go:117] "RemoveContainer" containerID="6441ed391fbd74bf98c617bc0cb92678dfdf5e67f889b06b5d5c65ad0f62af67" Jul 10 22:54:35.463564 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:35.463531 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:54:35.463642 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:35.463618 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:54:35.463680 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:35.463669 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:54:36.468702 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:36.468669 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" event={"ID":"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b","Type":"ContainerStarted","Data":"bcfcb8c7db9c547d671465fb7a7dd8c1cfabd61db12ed4e3dcf32fa9a08da589"} Jul 10 22:54:36.608237 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:36.608202 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:36.611847 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:36.611823 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:37.474108 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:37.474082 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:37.474950 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:37.474930 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:38.201092 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:38.201054 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:38.263780 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:38.263743 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:38.435116 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:38.435078 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs"] Jul 10 22:54:38.448430 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:38.448396 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:38.478939 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:38.478905 2565 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" secret="" err="secret \"e2e-mt-26b61da1-openshift-default-dockercfg-rh2d8\" not found" Jul 10 22:54:38.678477 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:38.678441 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:38.801177 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:38.801089 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:38.899700 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:38.899666 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:39.021599 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:39.021565 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:39.483278 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:39.483231 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" podUID="f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" containerName="istio-proxy" containerID="cri-o://bcfcb8c7db9c547d671465fb7a7dd8c1cfabd61db12ed4e3dcf32fa9a08da589" gracePeriod=30 Jul 10 22:54:44.584368 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.584332 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:44.646315 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.646281 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:44.732137 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.732106 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:44.747122 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.747063 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:44.786235 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.786206 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:44.851337 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.851301 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-envoy\") pod \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " Jul 10 22:54:44.851505 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.851351 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istiod-ca-cert\") pod \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " Jul 10 22:54:44.851505 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.851402 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-data\") pod \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " Jul 10 22:54:44.851505 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.851449 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-token\") pod \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " Jul 10 22:54:44.851505 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.851501 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-workload-certs\") pod \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " Jul 10 22:54:44.851714 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.851531 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-workload-socket\") pod \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " Jul 10 22:54:44.851714 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.851576 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-podinfo\") pod \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " Jul 10 22:54:44.851714 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.851604 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjgf7\" (UniqueName: \"kubernetes.io/projected/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-kube-api-access-qjgf7\") pod \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " Jul 10 22:54:44.851714 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.851631 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-credential-socket\") pod \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\" (UID: \"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b\") " Jul 10 22:54:44.851957 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.851854 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" (UID: "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:54:44.852046 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.852011 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" (UID: "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:54:44.852101 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.852041 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" (UID: "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:54:44.852101 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.852077 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istiod-ca-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:54:44.852223 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.852201 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" (UID: "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:54:44.852436 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.852405 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-data" (OuterVolumeSpecName: "istio-data") pod "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" (UID: "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:54:44.853766 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.853741 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" (UID: "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:54:44.853892 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.853764 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" (UID: "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:54:44.853937 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.853905 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" (UID: "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 22:54:44.853998 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.853980 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-kube-api-access-qjgf7" (OuterVolumeSpecName: "kube-api-access-qjgf7") pod "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" (UID: "f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b"). InnerVolumeSpecName "kube-api-access-qjgf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:54:44.930707 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.930675 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:44.952728 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.952703 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-token\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:54:44.952728 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.952722 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-workload-certs\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:54:44.952728 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.952732 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-workload-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:54:44.952920 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.952740 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-podinfo\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:54:44.952920 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.952750 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjgf7\" (UniqueName: \"kubernetes.io/projected/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-kube-api-access-qjgf7\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:54:44.952920 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.952760 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-credential-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:54:44.952920 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.952768 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-envoy\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:54:44.952920 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:44.952776 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b-istio-data\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:54:45.515231 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:45.515200 2565 generic.go:356] "Generic (PLEG): container finished" podID="f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" containerID="bcfcb8c7db9c547d671465fb7a7dd8c1cfabd61db12ed4e3dcf32fa9a08da589" exitCode=0 Jul 10 22:54:45.515410 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:45.515253 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" Jul 10 22:54:45.515410 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:45.515294 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" event={"ID":"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b","Type":"ContainerDied","Data":"bcfcb8c7db9c547d671465fb7a7dd8c1cfabd61db12ed4e3dcf32fa9a08da589"} Jul 10 22:54:45.515410 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:45.515353 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs" event={"ID":"f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b","Type":"ContainerDied","Data":"5db77a40fe6c47fd2ffcaa381fdd7a680bcee52c85fb4636661cdc2fdc64d1fe"} Jul 10 22:54:45.515410 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:45.515377 2565 scope.go:117] "RemoveContainer" containerID="bcfcb8c7db9c547d671465fb7a7dd8c1cfabd61db12ed4e3dcf32fa9a08da589" Jul 10 22:54:45.524538 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:45.524521 2565 scope.go:117] "RemoveContainer" containerID="6441ed391fbd74bf98c617bc0cb92678dfdf5e67f889b06b5d5c65ad0f62af67" Jul 10 22:54:45.532725 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:45.532708 2565 scope.go:117] "RemoveContainer" containerID="bcfcb8c7db9c547d671465fb7a7dd8c1cfabd61db12ed4e3dcf32fa9a08da589" Jul 10 22:54:45.533005 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:54:45.532987 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcfcb8c7db9c547d671465fb7a7dd8c1cfabd61db12ed4e3dcf32fa9a08da589\": container with ID starting with bcfcb8c7db9c547d671465fb7a7dd8c1cfabd61db12ed4e3dcf32fa9a08da589 not found: ID does not exist" containerID="bcfcb8c7db9c547d671465fb7a7dd8c1cfabd61db12ed4e3dcf32fa9a08da589" Jul 10 22:54:45.533069 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:45.533013 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcfcb8c7db9c547d671465fb7a7dd8c1cfabd61db12ed4e3dcf32fa9a08da589"} err="failed to get container status \"bcfcb8c7db9c547d671465fb7a7dd8c1cfabd61db12ed4e3dcf32fa9a08da589\": rpc error: code = NotFound desc = could not find container \"bcfcb8c7db9c547d671465fb7a7dd8c1cfabd61db12ed4e3dcf32fa9a08da589\": container with ID starting with bcfcb8c7db9c547d671465fb7a7dd8c1cfabd61db12ed4e3dcf32fa9a08da589 not found: ID does not exist" Jul 10 22:54:45.533069 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:45.533029 2565 scope.go:117] "RemoveContainer" containerID="6441ed391fbd74bf98c617bc0cb92678dfdf5e67f889b06b5d5c65ad0f62af67" Jul 10 22:54:45.533265 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:54:45.533245 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6441ed391fbd74bf98c617bc0cb92678dfdf5e67f889b06b5d5c65ad0f62af67\": container with ID starting with 6441ed391fbd74bf98c617bc0cb92678dfdf5e67f889b06b5d5c65ad0f62af67 not found: ID does not exist" containerID="6441ed391fbd74bf98c617bc0cb92678dfdf5e67f889b06b5d5c65ad0f62af67" Jul 10 22:54:45.533310 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:45.533271 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6441ed391fbd74bf98c617bc0cb92678dfdf5e67f889b06b5d5c65ad0f62af67"} err="failed to get container status \"6441ed391fbd74bf98c617bc0cb92678dfdf5e67f889b06b5d5c65ad0f62af67\": rpc error: code = NotFound desc = could not find container \"6441ed391fbd74bf98c617bc0cb92678dfdf5e67f889b06b5d5c65ad0f62af67\": container with ID starting with 6441ed391fbd74bf98c617bc0cb92678dfdf5e67f889b06b5d5c65ad0f62af67 not found: ID does not exist" Jul 10 22:54:45.536387 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:45.536366 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs"] Jul 10 22:54:45.542619 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:45.542599 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-26b61da1-openshift-default-57575d6499-mm4rs"] Jul 10 22:54:45.690877 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:45.690836 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" path="/var/lib/kubelet/pods/f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b/volumes" Jul 10 22:54:50.502577 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:50.502541 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:50.555269 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:50.555237 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:57.257750 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.257721 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:57.539646 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.539571 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz"] Jul 10 22:54:57.540032 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.540011 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc7fab5-7f5c-489c-a6da-77b459ddd036" containerName="maas-api" Jul 10 22:54:57.540182 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.540075 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc7fab5-7f5c-489c-a6da-77b459ddd036" containerName="maas-api" Jul 10 22:54:57.540182 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.540111 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" containerName="istio-proxy" Jul 10 22:54:57.540182 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.540121 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" containerName="istio-proxy" Jul 10 22:54:57.540182 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.540156 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" containerName="istio-proxy" Jul 10 22:54:57.540182 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.540166 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" containerName="istio-proxy" Jul 10 22:54:57.540424 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.540297 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bc7fab5-7f5c-489c-a6da-77b459ddd036" containerName="maas-api" Jul 10 22:54:57.540424 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.540315 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" containerName="istio-proxy" Jul 10 22:54:57.540424 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.540325 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5fcefb4-13d3-4e1a-aae6-9ca586d9cc0b" containerName="istio-proxy" Jul 10 22:54:57.543580 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.543557 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.545903 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.545882 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-727180d9-openshift-default-dockercfg-lcgmf\"" Jul 10 22:54:57.554576 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.554554 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz"] Jul 10 22:54:57.567036 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.567007 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:57.665216 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.665179 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-credential-socket\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.665398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.665225 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-podinfo\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.665398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.665299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-token\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.665398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.665327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-envoy\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.665398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.665349 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-workload-certs\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.665575 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.665449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjwhk\" (UniqueName: \"kubernetes.io/projected/0a59ddf4-76de-445d-b06d-dfe71f78f190-kube-api-access-hjwhk\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.665575 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.665483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a59ddf4-76de-445d-b06d-dfe71f78f190-istiod-ca-cert\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.665575 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.665529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-data\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.665575 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.665560 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-workload-socket\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.706950 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.706915 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:54:57.766168 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.766125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-credential-socket\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.766358 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.766178 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-podinfo\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.766358 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.766228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-token\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.766358 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.766264 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-envoy\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.766358 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.766322 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-workload-certs\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.766591 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.766396 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjwhk\" (UniqueName: \"kubernetes.io/projected/0a59ddf4-76de-445d-b06d-dfe71f78f190-kube-api-access-hjwhk\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.766591 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.766428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a59ddf4-76de-445d-b06d-dfe71f78f190-istiod-ca-cert\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.766591 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.766485 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-data\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.766591 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.766529 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-workload-socket\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.766591 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.766553 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-credential-socket\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.766884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.766844 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-workload-certs\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.766942 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.766902 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-workload-socket\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.767106 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.767083 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-data\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.767299 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.767273 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a59ddf4-76de-445d-b06d-dfe71f78f190-istiod-ca-cert\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.769247 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.769218 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-envoy\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.769498 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.769453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-podinfo\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.774519 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.774500 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-token\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.774612 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.774582 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjwhk\" (UniqueName: \"kubernetes.io/projected/0a59ddf4-76de-445d-b06d-dfe71f78f190-kube-api-access-hjwhk\") pod \"e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.854620 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.854531 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:57.991158 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.991131 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz"] Jul 10 22:54:57.992860 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:54:57.992833 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a59ddf4_76de_445d_b06d_dfe71f78f190.slice/crio-8619bf7d6da46a21951564dc70b2639c3f464c41949547701bbfae079929bca5 WatchSource:0}: Error finding container 8619bf7d6da46a21951564dc70b2639c3f464c41949547701bbfae079929bca5: Status 404 returned error can't find the container with id 8619bf7d6da46a21951564dc70b2639c3f464c41949547701bbfae079929bca5 Jul 10 22:54:57.994966 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.994925 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:54:57.995060 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.994992 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:54:57.995060 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:57.995018 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:54:58.587499 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:58.587466 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" event={"ID":"0a59ddf4-76de-445d-b06d-dfe71f78f190","Type":"ContainerStarted","Data":"418b7feb4daccb97236473b45de6f3ac6fbaa0e7faf47d564d53500e7fc7907a"} Jul 10 22:54:58.587916 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:58.587518 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" event={"ID":"0a59ddf4-76de-445d-b06d-dfe71f78f190","Type":"ContainerStarted","Data":"8619bf7d6da46a21951564dc70b2639c3f464c41949547701bbfae079929bca5"} Jul 10 22:54:58.608102 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:58.608050 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" podStartSLOduration=1.6080332259999999 podStartE2EDuration="1.608033226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:54:58.605917131 +0000 UTC m=+1962.438801513" watchObservedRunningTime="2026-07-10 22:54:58.608033226 +0000 UTC m=+1962.440917608" Jul 10 22:54:58.855264 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:58.855186 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:54:59.860357 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:54:59.860327 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:55:00.483497 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.483457 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:55:00.583661 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.583626 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:55:00.597804 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.597775 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:55:00.598908 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.598888 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:55:00.660207 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.660176 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4"] Jul 10 22:55:00.664169 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.664145 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:55:00.666360 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.666336 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-727180d9-serving-cert\"" Jul 10 22:55:00.672165 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.672142 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4"] Jul 10 22:55:00.698740 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.698705 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:55:00.792731 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.792639 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zp8t\" (UniqueName: \"kubernetes.io/projected/e35170a7-eefd-43da-bb82-ca3ef6ddb235-kube-api-access-2zp8t\") pod \"maas-api-e2e-mt-727180d9-7b749587f7-b2hv4\" (UID: \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:55:00.792731 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.792689 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e35170a7-eefd-43da-bb82-ca3ef6ddb235-maas-api-tls\") pod \"maas-api-e2e-mt-727180d9-7b749587f7-b2hv4\" (UID: \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:55:00.793031 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.792893 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35170a7-eefd-43da-bb82-ca3ef6ddb235-openshift-service-ca\") pod \"maas-api-e2e-mt-727180d9-7b749587f7-b2hv4\" (UID: \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:55:00.893746 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.893711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zp8t\" (UniqueName: \"kubernetes.io/projected/e35170a7-eefd-43da-bb82-ca3ef6ddb235-kube-api-access-2zp8t\") pod \"maas-api-e2e-mt-727180d9-7b749587f7-b2hv4\" (UID: \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:55:00.893746 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.893749 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e35170a7-eefd-43da-bb82-ca3ef6ddb235-maas-api-tls\") pod \"maas-api-e2e-mt-727180d9-7b749587f7-b2hv4\" (UID: \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:55:00.894251 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.893851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35170a7-eefd-43da-bb82-ca3ef6ddb235-openshift-service-ca\") pod \"maas-api-e2e-mt-727180d9-7b749587f7-b2hv4\" (UID: \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:55:00.894463 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.894444 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35170a7-eefd-43da-bb82-ca3ef6ddb235-openshift-service-ca\") pod \"maas-api-e2e-mt-727180d9-7b749587f7-b2hv4\" (UID: \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:55:00.896298 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.896273 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e35170a7-eefd-43da-bb82-ca3ef6ddb235-maas-api-tls\") pod \"maas-api-e2e-mt-727180d9-7b749587f7-b2hv4\" (UID: \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:55:00.901939 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.901914 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zp8t\" (UniqueName: \"kubernetes.io/projected/e35170a7-eefd-43da-bb82-ca3ef6ddb235-kube-api-access-2zp8t\") pod \"maas-api-e2e-mt-727180d9-7b749587f7-b2hv4\" (UID: \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:55:00.909595 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.909567 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:55:00.978391 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:00.978354 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:55:01.114932 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:01.113945 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4"] Jul 10 22:55:01.276448 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:01.276417 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:55:01.375349 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:01.375266 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:55:01.604866 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:01.604833 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" event={"ID":"e35170a7-eefd-43da-bb82-ca3ef6ddb235","Type":"ContainerStarted","Data":"c74e7aa4d74b11e29806dd52fd50a6fe1b51bfe8699b3b8eb57be3485c907559"} Jul 10 22:55:01.605064 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:01.604891 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" event={"ID":"e35170a7-eefd-43da-bb82-ca3ef6ddb235","Type":"ContainerStarted","Data":"9b2c19f9f6d9a611f1856ef545e1de169dbf783aaaf0df76f99affb552fc96af"} Jul 10 22:55:01.605242 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:01.605221 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:55:01.627088 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:01.626976 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" podStartSLOduration=1.23057911 podStartE2EDuration="1.626960582s" totalImagesPullingTime="396.381472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:55:01.624362415 +0000 UTC m=+1965.457246797" watchObservedRunningTime="2026-07-10 22:55:01.626960582 +0000 UTC m=+1965.459844965" Jul 10 22:55:04.976196 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:04.976161 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:55:08.620838 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:08.620783 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:55:11.731907 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:11.731875 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:55:11.834771 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:11.834732 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:55:14.685666 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:14.685631 2565 generic.go:356] "Generic (PLEG): container finished" podID="0a59ddf4-76de-445d-b06d-dfe71f78f190" containerID="418b7feb4daccb97236473b45de6f3ac6fbaa0e7faf47d564d53500e7fc7907a" exitCode=0 Jul 10 22:55:14.686077 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:14.685686 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" event={"ID":"0a59ddf4-76de-445d-b06d-dfe71f78f190","Type":"ContainerDied","Data":"418b7feb4daccb97236473b45de6f3ac6fbaa0e7faf47d564d53500e7fc7907a"} Jul 10 22:55:14.686077 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:14.685980 2565 scope.go:117] "RemoveContainer" containerID="418b7feb4daccb97236473b45de6f3ac6fbaa0e7faf47d564d53500e7fc7907a" Jul 10 22:55:14.686901 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:14.686869 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:55:14.686973 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:14.686956 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:55:14.687021 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:14.687004 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:55:15.598757 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:15.598722 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:55:15.694676 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:15.694642 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" event={"ID":"0a59ddf4-76de-445d-b06d-dfe71f78f190","Type":"ContainerStarted","Data":"454dd622a6f0cb946df98c39d71f9c47634dd5a4a9eb36bba1999861c7c4c162"} Jul 10 22:55:15.855560 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:15.855477 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:55:15.857040 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:15.857000 2565 patch_prober.go:28] interesting pod/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 10 22:55:15.857424 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:15.857053 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" podUID="0a59ddf4-76de-445d-b06d-dfe71f78f190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 10 22:55:16.855834 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:16.855772 2565 patch_prober.go:28] interesting pod/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 10 22:55:16.856253 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:16.855881 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" podUID="0a59ddf4-76de-445d-b06d-dfe71f78f190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 10 22:55:17.855760 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:17.855717 2565 patch_prober.go:28] interesting pod/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 10 22:55:17.855963 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:17.855785 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" podUID="0a59ddf4-76de-445d-b06d-dfe71f78f190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 10 22:55:18.855530 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:18.855438 2565 patch_prober.go:28] interesting pod/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 10 22:55:18.855530 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:18.855496 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" podUID="0a59ddf4-76de-445d-b06d-dfe71f78f190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 10 22:55:19.859091 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:19.859059 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:55:19.859551 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:19.859342 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:55:19.860134 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:19.860114 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:55:36.627398 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:36.627351 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:55:56.847417 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:55:56.847355 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:02.912493 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:02.912461 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4"] Jul 10 22:56:02.913017 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:02.912705 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" podUID="e35170a7-eefd-43da-bb82-ca3ef6ddb235" containerName="maas-api" containerID="cri-o://c74e7aa4d74b11e29806dd52fd50a6fe1b51bfe8699b3b8eb57be3485c907559" gracePeriod=30 Jul 10 22:56:02.957006 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:02.956966 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:03.038809 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.038749 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:03.168987 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.168932 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:56:03.249972 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.249943 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35170a7-eefd-43da-bb82-ca3ef6ddb235-openshift-service-ca\") pod \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\" (UID: \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\") " Jul 10 22:56:03.250138 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.249986 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zp8t\" (UniqueName: \"kubernetes.io/projected/e35170a7-eefd-43da-bb82-ca3ef6ddb235-kube-api-access-2zp8t\") pod \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\" (UID: \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\") " Jul 10 22:56:03.250216 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.250196 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e35170a7-eefd-43da-bb82-ca3ef6ddb235-maas-api-tls\") pod \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\" (UID: \"e35170a7-eefd-43da-bb82-ca3ef6ddb235\") " Jul 10 22:56:03.250303 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.250280 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e35170a7-eefd-43da-bb82-ca3ef6ddb235-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e35170a7-eefd-43da-bb82-ca3ef6ddb235" (UID: "e35170a7-eefd-43da-bb82-ca3ef6ddb235"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:56:03.250480 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.250464 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35170a7-eefd-43da-bb82-ca3ef6ddb235-openshift-service-ca\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:56:03.252053 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.252030 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35170a7-eefd-43da-bb82-ca3ef6ddb235-kube-api-access-2zp8t" (OuterVolumeSpecName: "kube-api-access-2zp8t") pod "e35170a7-eefd-43da-bb82-ca3ef6ddb235" (UID: "e35170a7-eefd-43da-bb82-ca3ef6ddb235"). InnerVolumeSpecName "kube-api-access-2zp8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:56:03.252183 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.252167 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e35170a7-eefd-43da-bb82-ca3ef6ddb235-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e35170a7-eefd-43da-bb82-ca3ef6ddb235" (UID: "e35170a7-eefd-43da-bb82-ca3ef6ddb235"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:56:03.351343 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.351316 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zp8t\" (UniqueName: \"kubernetes.io/projected/e35170a7-eefd-43da-bb82-ca3ef6ddb235-kube-api-access-2zp8t\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:56:03.351343 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.351340 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e35170a7-eefd-43da-bb82-ca3ef6ddb235-maas-api-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:56:03.897127 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.897095 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:03.940374 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.940339 2565 generic.go:356] "Generic (PLEG): container finished" podID="e35170a7-eefd-43da-bb82-ca3ef6ddb235" containerID="c74e7aa4d74b11e29806dd52fd50a6fe1b51bfe8699b3b8eb57be3485c907559" exitCode=0 Jul 10 22:56:03.940758 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.940388 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" Jul 10 22:56:03.940758 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.940393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" event={"ID":"e35170a7-eefd-43da-bb82-ca3ef6ddb235","Type":"ContainerDied","Data":"c74e7aa4d74b11e29806dd52fd50a6fe1b51bfe8699b3b8eb57be3485c907559"} Jul 10 22:56:03.940758 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.940448 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4" event={"ID":"e35170a7-eefd-43da-bb82-ca3ef6ddb235","Type":"ContainerDied","Data":"9b2c19f9f6d9a611f1856ef545e1de169dbf783aaaf0df76f99affb552fc96af"} Jul 10 22:56:03.940758 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.940480 2565 scope.go:117] "RemoveContainer" containerID="c74e7aa4d74b11e29806dd52fd50a6fe1b51bfe8699b3b8eb57be3485c907559" Jul 10 22:56:03.949033 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.949018 2565 scope.go:117] "RemoveContainer" containerID="c74e7aa4d74b11e29806dd52fd50a6fe1b51bfe8699b3b8eb57be3485c907559" Jul 10 22:56:03.949278 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:56:03.949259 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74e7aa4d74b11e29806dd52fd50a6fe1b51bfe8699b3b8eb57be3485c907559\": container with ID starting with c74e7aa4d74b11e29806dd52fd50a6fe1b51bfe8699b3b8eb57be3485c907559 not found: ID does not exist" containerID="c74e7aa4d74b11e29806dd52fd50a6fe1b51bfe8699b3b8eb57be3485c907559" Jul 10 22:56:03.949352 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.949298 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74e7aa4d74b11e29806dd52fd50a6fe1b51bfe8699b3b8eb57be3485c907559"} err="failed to get container status \"c74e7aa4d74b11e29806dd52fd50a6fe1b51bfe8699b3b8eb57be3485c907559\": rpc error: code = NotFound desc = could not find container \"c74e7aa4d74b11e29806dd52fd50a6fe1b51bfe8699b3b8eb57be3485c907559\": container with ID starting with c74e7aa4d74b11e29806dd52fd50a6fe1b51bfe8699b3b8eb57be3485c907559 not found: ID does not exist" Jul 10 22:56:03.956312 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.956286 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4"] Jul 10 22:56:03.959355 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:03.959336 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-727180d9-7b749587f7-b2hv4"] Jul 10 22:56:05.691615 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:05.691579 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e35170a7-eefd-43da-bb82-ca3ef6ddb235" path="/var/lib/kubelet/pods/e35170a7-eefd-43da-bb82-ca3ef6ddb235/volumes" Jul 10 22:56:08.977267 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:08.977212 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:42.806000 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:42.805920 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:45.693266 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:45.693234 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:45.863300 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:45.863235 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:51.391189 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:51.391150 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:51.491020 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:51.490982 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:51.661867 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:51.661761 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz"] Jul 10 22:56:51.662108 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:51.662079 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" podUID="0a59ddf4-76de-445d-b06d-dfe71f78f190" containerName="istio-proxy" containerID="cri-o://454dd622a6f0cb946df98c39d71f9c47634dd5a4a9eb36bba1999861c7c4c162" gracePeriod=30 Jul 10 22:56:51.684710 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:51.684678 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:51.911593 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:51.911563 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:53.010740 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:53.010707 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:53.099455 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:53.099424 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:53.902278 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:53.902237 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:53.999327 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:53.999285 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:54.099203 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:54.099169 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:55.448698 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:55.448649 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:56:56.915299 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:56.915271 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:56:57.005960 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.005916 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-credential-socket\") pod \"0a59ddf4-76de-445d-b06d-dfe71f78f190\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " Jul 10 22:56:57.005960 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.005969 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-envoy\") pod \"0a59ddf4-76de-445d-b06d-dfe71f78f190\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " Jul 10 22:56:57.006238 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.006008 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-workload-certs\") pod \"0a59ddf4-76de-445d-b06d-dfe71f78f190\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " Jul 10 22:56:57.006238 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.006024 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-workload-socket\") pod \"0a59ddf4-76de-445d-b06d-dfe71f78f190\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " Jul 10 22:56:57.006238 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.006047 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-podinfo\") pod \"0a59ddf4-76de-445d-b06d-dfe71f78f190\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " Jul 10 22:56:57.006238 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.006062 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-token\") pod \"0a59ddf4-76de-445d-b06d-dfe71f78f190\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " Jul 10 22:56:57.006238 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.006098 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjwhk\" (UniqueName: \"kubernetes.io/projected/0a59ddf4-76de-445d-b06d-dfe71f78f190-kube-api-access-hjwhk\") pod \"0a59ddf4-76de-445d-b06d-dfe71f78f190\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " Jul 10 22:56:57.006238 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.006167 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a59ddf4-76de-445d-b06d-dfe71f78f190-istiod-ca-cert\") pod \"0a59ddf4-76de-445d-b06d-dfe71f78f190\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " Jul 10 22:56:57.006238 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.006191 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-data\") pod \"0a59ddf4-76de-445d-b06d-dfe71f78f190\" (UID: \"0a59ddf4-76de-445d-b06d-dfe71f78f190\") " Jul 10 22:56:57.006238 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.006220 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0a59ddf4-76de-445d-b06d-dfe71f78f190" (UID: "0a59ddf4-76de-445d-b06d-dfe71f78f190"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:56:57.006631 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.006395 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0a59ddf4-76de-445d-b06d-dfe71f78f190" (UID: "0a59ddf4-76de-445d-b06d-dfe71f78f190"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:56:57.006631 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.006448 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-credential-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:56:57.006631 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.006499 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0a59ddf4-76de-445d-b06d-dfe71f78f190" (UID: "0a59ddf4-76de-445d-b06d-dfe71f78f190"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:56:57.006875 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.006664 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a59ddf4-76de-445d-b06d-dfe71f78f190-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0a59ddf4-76de-445d-b06d-dfe71f78f190" (UID: "0a59ddf4-76de-445d-b06d-dfe71f78f190"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:56:57.007463 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.007434 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-data" (OuterVolumeSpecName: "istio-data") pod "0a59ddf4-76de-445d-b06d-dfe71f78f190" (UID: "0a59ddf4-76de-445d-b06d-dfe71f78f190"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:56:57.008395 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.008372 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0a59ddf4-76de-445d-b06d-dfe71f78f190" (UID: "0a59ddf4-76de-445d-b06d-dfe71f78f190"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 22:56:57.008513 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.008396 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a59ddf4-76de-445d-b06d-dfe71f78f190-kube-api-access-hjwhk" (OuterVolumeSpecName: "kube-api-access-hjwhk") pod "0a59ddf4-76de-445d-b06d-dfe71f78f190" (UID: "0a59ddf4-76de-445d-b06d-dfe71f78f190"). InnerVolumeSpecName "kube-api-access-hjwhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:56:57.008513 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.008488 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0a59ddf4-76de-445d-b06d-dfe71f78f190" (UID: "0a59ddf4-76de-445d-b06d-dfe71f78f190"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:56:57.008513 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.008509 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-token" (OuterVolumeSpecName: "istio-token") pod "0a59ddf4-76de-445d-b06d-dfe71f78f190" (UID: "0a59ddf4-76de-445d-b06d-dfe71f78f190"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:56:57.106855 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.106826 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-workload-certs\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:56:57.106855 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.106849 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-workload-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:56:57.106855 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.106858 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-podinfo\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:56:57.107060 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.106866 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-token\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:56:57.107060 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.106874 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjwhk\" (UniqueName: \"kubernetes.io/projected/0a59ddf4-76de-445d-b06d-dfe71f78f190-kube-api-access-hjwhk\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:56:57.107060 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.106883 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a59ddf4-76de-445d-b06d-dfe71f78f190-istiod-ca-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:56:57.107060 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.106891 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-data\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:56:57.107060 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.106898 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a59ddf4-76de-445d-b06d-dfe71f78f190-istio-envoy\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:56:57.220335 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.220302 2565 generic.go:356] "Generic (PLEG): container finished" podID="0a59ddf4-76de-445d-b06d-dfe71f78f190" containerID="454dd622a6f0cb946df98c39d71f9c47634dd5a4a9eb36bba1999861c7c4c162" exitCode=0 Jul 10 22:56:57.220500 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.220344 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" Jul 10 22:56:57.220500 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.220353 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" event={"ID":"0a59ddf4-76de-445d-b06d-dfe71f78f190","Type":"ContainerDied","Data":"454dd622a6f0cb946df98c39d71f9c47634dd5a4a9eb36bba1999861c7c4c162"} Jul 10 22:56:57.220500 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.220397 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz" event={"ID":"0a59ddf4-76de-445d-b06d-dfe71f78f190","Type":"ContainerDied","Data":"8619bf7d6da46a21951564dc70b2639c3f464c41949547701bbfae079929bca5"} Jul 10 22:56:57.220500 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.220419 2565 scope.go:117] "RemoveContainer" containerID="454dd622a6f0cb946df98c39d71f9c47634dd5a4a9eb36bba1999861c7c4c162" Jul 10 22:56:57.230172 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.230155 2565 scope.go:117] "RemoveContainer" containerID="418b7feb4daccb97236473b45de6f3ac6fbaa0e7faf47d564d53500e7fc7907a" Jul 10 22:56:57.238393 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.238376 2565 scope.go:117] "RemoveContainer" containerID="454dd622a6f0cb946df98c39d71f9c47634dd5a4a9eb36bba1999861c7c4c162" Jul 10 22:56:57.238642 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:56:57.238622 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454dd622a6f0cb946df98c39d71f9c47634dd5a4a9eb36bba1999861c7c4c162\": container with ID starting with 454dd622a6f0cb946df98c39d71f9c47634dd5a4a9eb36bba1999861c7c4c162 not found: ID does not exist" containerID="454dd622a6f0cb946df98c39d71f9c47634dd5a4a9eb36bba1999861c7c4c162" Jul 10 22:56:57.238705 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.238654 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454dd622a6f0cb946df98c39d71f9c47634dd5a4a9eb36bba1999861c7c4c162"} err="failed to get container status \"454dd622a6f0cb946df98c39d71f9c47634dd5a4a9eb36bba1999861c7c4c162\": rpc error: code = NotFound desc = could not find container \"454dd622a6f0cb946df98c39d71f9c47634dd5a4a9eb36bba1999861c7c4c162\": container with ID starting with 454dd622a6f0cb946df98c39d71f9c47634dd5a4a9eb36bba1999861c7c4c162 not found: ID does not exist" Jul 10 22:56:57.238705 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.238680 2565 scope.go:117] "RemoveContainer" containerID="418b7feb4daccb97236473b45de6f3ac6fbaa0e7faf47d564d53500e7fc7907a" Jul 10 22:56:57.238969 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:56:57.238951 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418b7feb4daccb97236473b45de6f3ac6fbaa0e7faf47d564d53500e7fc7907a\": container with ID starting with 418b7feb4daccb97236473b45de6f3ac6fbaa0e7faf47d564d53500e7fc7907a not found: ID does not exist" containerID="418b7feb4daccb97236473b45de6f3ac6fbaa0e7faf47d564d53500e7fc7907a" Jul 10 22:56:57.239015 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.238975 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418b7feb4daccb97236473b45de6f3ac6fbaa0e7faf47d564d53500e7fc7907a"} err="failed to get container status \"418b7feb4daccb97236473b45de6f3ac6fbaa0e7faf47d564d53500e7fc7907a\": rpc error: code = NotFound desc = could not find container \"418b7feb4daccb97236473b45de6f3ac6fbaa0e7faf47d564d53500e7fc7907a\": container with ID starting with 418b7feb4daccb97236473b45de6f3ac6fbaa0e7faf47d564d53500e7fc7907a not found: ID does not exist" Jul 10 22:56:57.242635 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.242613 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz"] Jul 10 22:56:57.246370 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.246348 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-727180d9-openshift-default-74b45d96f5-xdqxz"] Jul 10 22:56:57.697232 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:56:57.697196 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a59ddf4-76de-445d-b06d-dfe71f78f190" path="/var/lib/kubelet/pods/0a59ddf4-76de-445d-b06d-dfe71f78f190/volumes" Jul 10 22:57:01.313112 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:01.313062 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:01.413101 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:01.413068 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:02.002068 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:02.002030 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:07.638965 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:07.638931 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:07.704768 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:07.704718 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:07.786627 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:07.786595 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:07.892821 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:07.892700 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:18.323382 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:18.323333 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:18.402487 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:18.402457 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:57:18.408229 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:18.408209 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 22:57:18.665071 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:18.664948 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:39.631904 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:39.631865 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:45.494365 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:45.494330 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:45.621713 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:45.621678 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:45.854181 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:45.854076 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:46.130125 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.130042 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs"] Jul 10 22:57:46.130493 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.130474 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a59ddf4-76de-445d-b06d-dfe71f78f190" containerName="istio-proxy" Jul 10 22:57:46.130579 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.130495 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a59ddf4-76de-445d-b06d-dfe71f78f190" containerName="istio-proxy" Jul 10 22:57:46.130579 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.130523 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e35170a7-eefd-43da-bb82-ca3ef6ddb235" containerName="maas-api" Jul 10 22:57:46.130579 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.130532 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e35170a7-eefd-43da-bb82-ca3ef6ddb235" containerName="maas-api" Jul 10 22:57:46.130737 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.130613 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e35170a7-eefd-43da-bb82-ca3ef6ddb235" containerName="maas-api" Jul 10 22:57:46.130737 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.130626 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a59ddf4-76de-445d-b06d-dfe71f78f190" containerName="istio-proxy" Jul 10 22:57:46.130737 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.130636 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a59ddf4-76de-445d-b06d-dfe71f78f190" containerName="istio-proxy" Jul 10 22:57:46.133944 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.133923 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.136706 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.136683 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c8c45c-openshift-default-dockercfg-dkmc7\"" Jul 10 22:57:46.146528 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.146501 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs"] Jul 10 22:57:46.157621 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.157591 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:46.216484 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.216433 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-credential-socket\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.216767 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.216743 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-workload-socket\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.216895 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.216813 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-workload-certs\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.216895 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.216842 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f25824-9f93-4530-9bb6-8636686a1971-istiod-ca-cert\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.216895 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.216880 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f25824-9f93-4530-9bb6-8636686a1971-istio-token\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.217064 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.216908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd9n6\" (UniqueName: \"kubernetes.io/projected/f6f25824-9f93-4530-9bb6-8636686a1971-kube-api-access-dd9n6\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.217064 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.216947 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-istio-data\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.217064 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.217003 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-istio-envoy\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.217064 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.217037 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f25824-9f93-4530-9bb6-8636686a1971-istio-podinfo\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.317568 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.317536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-credential-socket\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.317776 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.317583 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-workload-socket\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.317776 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.317608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-workload-certs\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.317776 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.317633 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f25824-9f93-4530-9bb6-8636686a1971-istiod-ca-cert\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.317776 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.317658 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f25824-9f93-4530-9bb6-8636686a1971-istio-token\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.317776 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.317676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd9n6\" (UniqueName: \"kubernetes.io/projected/f6f25824-9f93-4530-9bb6-8636686a1971-kube-api-access-dd9n6\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.317776 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.317704 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-istio-data\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.318155 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.317750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-istio-envoy\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.318155 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.317862 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f25824-9f93-4530-9bb6-8636686a1971-istio-podinfo\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.318155 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.318012 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-credential-socket\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.318155 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.318111 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-istio-data\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.318363 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.318176 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-workload-certs\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.318363 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.318310 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-workload-socket\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.318431 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.318363 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f25824-9f93-4530-9bb6-8636686a1971-istiod-ca-cert\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.320129 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.320109 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-istio-envoy\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.320433 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.320413 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f25824-9f93-4530-9bb6-8636686a1971-istio-podinfo\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.325392 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.325365 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f25824-9f93-4530-9bb6-8636686a1971-istio-token\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.325615 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.325597 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd9n6\" (UniqueName: \"kubernetes.io/projected/f6f25824-9f93-4530-9bb6-8636686a1971-kube-api-access-dd9n6\") pod \"e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.445114 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.445015 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:46.579290 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.579255 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:46.596067 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.596043 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs"] Jul 10 22:57:46.597931 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:57:46.597906 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f25824_9f93_4530_9bb6_8636686a1971.slice/crio-0404adbc14404735f0823c3ef16b6fc4d5add9352e0ddba361ef85ec5ad305a5 WatchSource:0}: Error finding container 0404adbc14404735f0823c3ef16b6fc4d5add9352e0ddba361ef85ec5ad305a5: Status 404 returned error can't find the container with id 0404adbc14404735f0823c3ef16b6fc4d5add9352e0ddba361ef85ec5ad305a5 Jul 10 22:57:46.600614 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.600575 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:57:46.600696 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.600674 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:57:46.600740 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:46.600728 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:57:47.487290 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:47.487250 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" event={"ID":"f6f25824-9f93-4530-9bb6-8636686a1971","Type":"ContainerStarted","Data":"efbe9355eb2ec742c6bf3daa8f868d9ae51503f5398a12dd6af3bece9a5f0456"} Jul 10 22:57:47.487445 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:47.487313 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" event={"ID":"f6f25824-9f93-4530-9bb6-8636686a1971","Type":"ContainerStarted","Data":"0404adbc14404735f0823c3ef16b6fc4d5add9352e0ddba361ef85ec5ad305a5"} Jul 10 22:57:47.508206 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:47.508154 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" podStartSLOduration=1.508136291 podStartE2EDuration="1.508136291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:57:47.505985829 +0000 UTC m=+2131.338870212" watchObservedRunningTime="2026-07-10 22:57:47.508136291 +0000 UTC m=+2131.341020673" Jul 10 22:57:48.445577 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:48.445533 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:48.450449 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:48.450424 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:48.492841 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:48.492815 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:48.493759 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:48.493737 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:57:49.421154 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.421117 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:49.520551 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.520514 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:49.614295 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.614256 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc"] Jul 10 22:57:49.614809 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.614774 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a59ddf4-76de-445d-b06d-dfe71f78f190" containerName="istio-proxy" Jul 10 22:57:49.614943 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.614823 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a59ddf4-76de-445d-b06d-dfe71f78f190" containerName="istio-proxy" Jul 10 22:57:49.618068 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.618048 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:49.620381 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.620362 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c8c45c-serving-cert\"" Jul 10 22:57:49.629247 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.629223 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc"] Jul 10 22:57:49.648615 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.648580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b104947a-8899-4694-906d-b531b0a06e4b-maas-api-tls\") pod \"maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc\" (UID: \"b104947a-8899-4694-906d-b531b0a06e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:49.648764 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.648647 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b104947a-8899-4694-906d-b531b0a06e4b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc\" (UID: \"b104947a-8899-4694-906d-b531b0a06e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:49.648764 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.648722 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vwzw\" (UniqueName: \"kubernetes.io/projected/b104947a-8899-4694-906d-b531b0a06e4b-kube-api-access-2vwzw\") pod \"maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc\" (UID: \"b104947a-8899-4694-906d-b531b0a06e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:49.653846 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.653757 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:49.750160 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.750112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vwzw\" (UniqueName: \"kubernetes.io/projected/b104947a-8899-4694-906d-b531b0a06e4b-kube-api-access-2vwzw\") pod \"maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc\" (UID: \"b104947a-8899-4694-906d-b531b0a06e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:49.750367 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.750205 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b104947a-8899-4694-906d-b531b0a06e4b-maas-api-tls\") pod \"maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc\" (UID: \"b104947a-8899-4694-906d-b531b0a06e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:49.750367 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.750249 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b104947a-8899-4694-906d-b531b0a06e4b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc\" (UID: \"b104947a-8899-4694-906d-b531b0a06e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:49.750507 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:57:49.750400 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-serving-cert: secret "maas-api-e2e-inf-a-c8c45c-serving-cert" not found Jul 10 22:57:49.750507 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:57:49.750495 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b104947a-8899-4694-906d-b531b0a06e4b-maas-api-tls podName:b104947a-8899-4694-906d-b531b0a06e4b nodeName:}" failed. No retries permitted until 2026-07-10 22:57:50.250471083 +0000 UTC m=+2134.083355463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b104947a-8899-4694-906d-b531b0a06e4b-maas-api-tls") pod "maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" (UID: "b104947a-8899-4694-906d-b531b0a06e4b") : secret "maas-api-e2e-inf-a-c8c45c-serving-cert" not found Jul 10 22:57:49.750861 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.750844 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b104947a-8899-4694-906d-b531b0a06e4b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc\" (UID: \"b104947a-8899-4694-906d-b531b0a06e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:49.759339 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.759315 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vwzw\" (UniqueName: \"kubernetes.io/projected/b104947a-8899-4694-906d-b531b0a06e4b-kube-api-access-2vwzw\") pod \"maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc\" (UID: \"b104947a-8899-4694-906d-b531b0a06e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:49.835689 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:49.835656 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:50.202987 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:50.202893 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:50.255402 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:50.255371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b104947a-8899-4694-906d-b531b0a06e4b-maas-api-tls\") pod \"maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc\" (UID: \"b104947a-8899-4694-906d-b531b0a06e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:50.257893 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:50.257868 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b104947a-8899-4694-906d-b531b0a06e4b-maas-api-tls\") pod \"maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc\" (UID: \"b104947a-8899-4694-906d-b531b0a06e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:50.306768 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:50.306726 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:50.530433 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:50.530394 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:50.671295 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:50.671266 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc"] Jul 10 22:57:50.673722 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:57:50.673689 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb104947a_8899_4694_906d_b531b0a06e4b.slice/crio-38127207c9e65798caa9e8d953265a9fb857858d8955213c92a8cf5789247e29 WatchSource:0}: Error finding container 38127207c9e65798caa9e8d953265a9fb857858d8955213c92a8cf5789247e29: Status 404 returned error can't find the container with id 38127207c9e65798caa9e8d953265a9fb857858d8955213c92a8cf5789247e29 Jul 10 22:57:50.675473 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:50.675452 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 22:57:51.510181 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:51.510138 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" event={"ID":"b104947a-8899-4694-906d-b531b0a06e4b","Type":"ContainerStarted","Data":"5dc0f6c15fe5c6fb5d9f8e77caff8d14e834d6554b84cba1d0b2cba850f93faa"} Jul 10 22:57:51.510349 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:51.510207 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" event={"ID":"b104947a-8899-4694-906d-b531b0a06e4b","Type":"ContainerStarted","Data":"38127207c9e65798caa9e8d953265a9fb857858d8955213c92a8cf5789247e29"} Jul 10 22:57:51.510349 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:51.510239 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:51.532026 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:51.531979 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" podStartSLOduration=2.063052214 podStartE2EDuration="2.531962283s" totalImagesPullingTime="468.910069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:57:51.530759307 +0000 UTC m=+2135.363643693" watchObservedRunningTime="2026-07-10 22:57:51.531962283 +0000 UTC m=+2135.364846702" Jul 10 22:57:53.922884 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:53.922848 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:54.259356 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.259317 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:54.532967 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.532864 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm"] Jul 10 22:57:54.536455 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.536433 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.539080 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.539044 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c49b05-openshift-default-dockercfg-7tcb4\"" Jul 10 22:57:54.550317 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.550275 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm"] Jul 10 22:57:54.555643 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.555601 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:54.596452 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.596409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-workload-certs\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.596452 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.596455 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmwt8\" (UniqueName: \"kubernetes.io/projected/f79792ff-8752-42c6-b14b-3fc648ddd168-kube-api-access-wmwt8\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.596705 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.596480 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-podinfo\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.596705 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.596591 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-data\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.596705 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.596631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-token\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.596705 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.596650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-workload-socket\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.596952 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.596707 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-envoy\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.596952 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.596758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-credential-socket\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.596952 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.596815 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79792ff-8752-42c6-b14b-3fc648ddd168-istiod-ca-cert\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.655287 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.655252 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:54.697826 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.697754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-data\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.697826 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.697832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-token\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.698073 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.697852 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-workload-socket\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.698073 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.697873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-envoy\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.698073 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.697897 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-credential-socket\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.698073 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.697928 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79792ff-8752-42c6-b14b-3fc648ddd168-istiod-ca-cert\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.698073 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.697993 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-workload-certs\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.698073 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.698019 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmwt8\" (UniqueName: \"kubernetes.io/projected/f79792ff-8752-42c6-b14b-3fc648ddd168-kube-api-access-wmwt8\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.698073 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.698054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-podinfo\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.698445 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.698261 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-data\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.698445 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.698290 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-workload-socket\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.698549 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.698517 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-credential-socket\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.698549 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.698534 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-workload-certs\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.699140 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.699115 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79792ff-8752-42c6-b14b-3fc648ddd168-istiod-ca-cert\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.700509 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.700487 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-envoy\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.700727 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.700707 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-podinfo\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.705419 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.705398 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmwt8\" (UniqueName: \"kubernetes.io/projected/f79792ff-8752-42c6-b14b-3fc648ddd168-kube-api-access-wmwt8\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.705543 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.705522 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-token\") pod \"e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.856844 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.856731 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:54.993715 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.993687 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm"] Jul 10 22:57:54.995422 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:57:54.995378 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79792ff_8752_42c6_b14b_3fc648ddd168.slice/crio-83351d703d3623621a75e9535cb26e58e83ed8ded6816905ba8dfc521a24f2eb WatchSource:0}: Error finding container 83351d703d3623621a75e9535cb26e58e83ed8ded6816905ba8dfc521a24f2eb: Status 404 returned error can't find the container with id 83351d703d3623621a75e9535cb26e58e83ed8ded6816905ba8dfc521a24f2eb Jul 10 22:57:54.997768 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.997735 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:57:54.997864 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.997823 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:57:54.997909 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:54.997861 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:57:55.533083 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:55.533050 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" event={"ID":"f79792ff-8752-42c6-b14b-3fc648ddd168","Type":"ContainerStarted","Data":"361600e0af086b0b4c40332e895aa4ae414b1614deca651751db01ca01548e63"} Jul 10 22:57:55.533267 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:55.533102 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" event={"ID":"f79792ff-8752-42c6-b14b-3fc648ddd168","Type":"ContainerStarted","Data":"83351d703d3623621a75e9535cb26e58e83ed8ded6816905ba8dfc521a24f2eb"} Jul 10 22:57:55.559089 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:55.559043 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" podStartSLOduration=1.5590273030000001 podStartE2EDuration="1.559027303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:57:55.557119437 +0000 UTC m=+2139.390003822" watchObservedRunningTime="2026-07-10 22:57:55.559027303 +0000 UTC m=+2139.391911685" Jul 10 22:57:55.856891 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:55.856803 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:56.861971 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:56.861941 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:57.441157 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.441093 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:57.523393 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.523362 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:57:57.541658 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.541608 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:57.544413 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.544388 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:57.545547 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.545530 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:57:57.627134 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.627097 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp"] Jul 10 22:57:57.631576 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.631551 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 22:57:57.634108 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.634082 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c49b05-serving-cert\"" Jul 10 22:57:57.641457 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.641430 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp"] Jul 10 22:57:57.672834 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.672778 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:57.728957 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.728917 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97059587-6725-432f-b7bf-a65226e08ecd-maas-api-tls\") pod \"maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp\" (UID: \"97059587-6725-432f-b7bf-a65226e08ecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 22:57:57.729153 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.729066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mkfn\" (UniqueName: \"kubernetes.io/projected/97059587-6725-432f-b7bf-a65226e08ecd-kube-api-access-2mkfn\") pod \"maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp\" (UID: \"97059587-6725-432f-b7bf-a65226e08ecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 22:57:57.729153 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.729097 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97059587-6725-432f-b7bf-a65226e08ecd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp\" (UID: \"97059587-6725-432f-b7bf-a65226e08ecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 22:57:57.772329 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.772279 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:57.830597 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.830397 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97059587-6725-432f-b7bf-a65226e08ecd-maas-api-tls\") pod \"maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp\" (UID: \"97059587-6725-432f-b7bf-a65226e08ecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 22:57:57.830597 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.830504 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mkfn\" (UniqueName: \"kubernetes.io/projected/97059587-6725-432f-b7bf-a65226e08ecd-kube-api-access-2mkfn\") pod \"maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp\" (UID: \"97059587-6725-432f-b7bf-a65226e08ecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 22:57:57.830597 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.830533 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97059587-6725-432f-b7bf-a65226e08ecd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp\" (UID: \"97059587-6725-432f-b7bf-a65226e08ecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 22:57:57.833061 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.833005 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97059587-6725-432f-b7bf-a65226e08ecd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp\" (UID: \"97059587-6725-432f-b7bf-a65226e08ecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 22:57:57.834999 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.834969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97059587-6725-432f-b7bf-a65226e08ecd-maas-api-tls\") pod \"maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp\" (UID: \"97059587-6725-432f-b7bf-a65226e08ecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 22:57:57.839641 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.839611 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mkfn\" (UniqueName: \"kubernetes.io/projected/97059587-6725-432f-b7bf-a65226e08ecd-kube-api-access-2mkfn\") pod \"maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp\" (UID: \"97059587-6725-432f-b7bf-a65226e08ecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 22:57:57.945802 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:57.945760 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 22:57:58.092727 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:58.092697 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp"] Jul 10 22:57:58.094764 ip-10-0-136-68 kubenswrapper[2565]: W0710 22:57:58.094736 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97059587_6725_432f_b7bf_a65226e08ecd.slice/crio-2faebce05cfde8d99c337acdeed325635d81c16530dc330e06c80907dba42e57 WatchSource:0}: Error finding container 2faebce05cfde8d99c337acdeed325635d81c16530dc330e06c80907dba42e57: Status 404 returned error can't find the container with id 2faebce05cfde8d99c337acdeed325635d81c16530dc330e06c80907dba42e57 Jul 10 22:57:58.098617 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:58.098593 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:58.201586 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:58.199927 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:57:58.553008 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:58.552964 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" event={"ID":"97059587-6725-432f-b7bf-a65226e08ecd","Type":"ContainerStarted","Data":"d032d401c8a467c1fba0836ca4b33447e6806cbbae23646fc3cfa2d573b94b21"} Jul 10 22:57:58.553203 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:58.553074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" event={"ID":"97059587-6725-432f-b7bf-a65226e08ecd","Type":"ContainerStarted","Data":"2faebce05cfde8d99c337acdeed325635d81c16530dc330e06c80907dba42e57"} Jul 10 22:57:58.553324 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:58.553299 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 22:57:58.572250 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:57:58.572184 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" podStartSLOduration=1.250486539 podStartE2EDuration="1.572168418s" totalImagesPullingTime="321.681879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:57:58.570102838 +0000 UTC m=+2142.402987220" watchObservedRunningTime="2026-07-10 22:57:58.572168418 +0000 UTC m=+2142.405052834" Jul 10 22:58:01.932974 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:01.932851 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:58:04.535077 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:04.535020 2565 patch_prober.go:28] interesting pod/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 22:58:04.535704 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:04.535660 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" podUID="f6f25824-9f93-4530-9bb6-8636686a1971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 22:58:04.565020 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:04.564986 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 22:58:05.595413 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:05.595326 2565 generic.go:356] "Generic (PLEG): container finished" podID="f6f25824-9f93-4530-9bb6-8636686a1971" containerID="efbe9355eb2ec742c6bf3daa8f868d9ae51503f5398a12dd6af3bece9a5f0456" exitCode=0 Jul 10 22:58:05.595413 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:05.595398 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" event={"ID":"f6f25824-9f93-4530-9bb6-8636686a1971","Type":"ContainerDied","Data":"efbe9355eb2ec742c6bf3daa8f868d9ae51503f5398a12dd6af3bece9a5f0456"} Jul 10 22:58:05.595885 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:05.595825 2565 scope.go:117] "RemoveContainer" containerID="efbe9355eb2ec742c6bf3daa8f868d9ae51503f5398a12dd6af3bece9a5f0456" Jul 10 22:58:05.596940 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:05.596899 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:58:05.597050 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:05.596979 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:58:05.597050 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:05.597019 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:58:06.602431 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:06.602391 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" event={"ID":"f6f25824-9f93-4530-9bb6-8636686a1971","Type":"ContainerStarted","Data":"cb07b7178f571aaa99eb5461d9b8d7d6b0226c13d55967aba8997486e0deb83a"} Jul 10 22:58:07.446308 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:07.446256 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:58:07.446527 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:07.446409 2565 patch_prober.go:28] interesting pod/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 10 22:58:07.446527 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:07.446471 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" podUID="f6f25824-9f93-4530-9bb6-8636686a1971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 10 22:58:08.445740 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:08.445630 2565 patch_prober.go:28] interesting pod/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 10 22:58:08.445740 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:08.445711 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" podUID="f6f25824-9f93-4530-9bb6-8636686a1971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 10 22:58:09.445925 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:09.445878 2565 patch_prober.go:28] interesting pod/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 10 22:58:09.446522 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:09.446485 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" podUID="f6f25824-9f93-4530-9bb6-8636686a1971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 10 22:58:09.810958 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:09.810858 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:58:10.446842 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:10.446337 2565 patch_prober.go:28] interesting pod/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 10 22:58:10.446842 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:10.446408 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" podUID="f6f25824-9f93-4530-9bb6-8636686a1971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 10 22:58:11.449208 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:11.449159 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:58:11.449731 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:11.449710 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:58:11.450504 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:11.450484 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 22:58:12.546158 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:12.546120 2565 patch_prober.go:28] interesting pod/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 10 22:58:12.546579 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:12.546175 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" podUID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 10 22:58:12.641248 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:12.641216 2565 generic.go:356] "Generic (PLEG): container finished" podID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerID="361600e0af086b0b4c40332e895aa4ae414b1614deca651751db01ca01548e63" exitCode=0 Jul 10 22:58:12.641426 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:12.641288 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" event={"ID":"f79792ff-8752-42c6-b14b-3fc648ddd168","Type":"ContainerDied","Data":"361600e0af086b0b4c40332e895aa4ae414b1614deca651751db01ca01548e63"} Jul 10 22:58:12.641891 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:12.641832 2565 scope.go:117] "RemoveContainer" containerID="361600e0af086b0b4c40332e895aa4ae414b1614deca651751db01ca01548e63" Jul 10 22:58:12.643080 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:12.643053 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:58:12.643176 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:12.643131 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:58:12.643176 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:12.643168 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 22:58:13.651010 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:13.650970 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" event={"ID":"f79792ff-8752-42c6-b14b-3fc648ddd168","Type":"ContainerStarted","Data":"40dacb91e8156f3aaabed395bbcb79503367e974a3203dd04d85767094ae0e6d"} Jul 10 22:58:13.857657 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:13.857616 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:58:13.857986 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:13.857961 2565 patch_prober.go:28] interesting pod/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 10 22:58:13.858037 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:13.858017 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" podUID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 10 22:58:14.858097 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:14.858056 2565 patch_prober.go:28] interesting pod/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 10 22:58:14.858497 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:14.858116 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" podUID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 10 22:58:15.858211 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:15.858170 2565 patch_prober.go:28] interesting pod/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 10 22:58:15.858580 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:15.858246 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" podUID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 10 22:58:16.858066 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:16.858023 2565 patch_prober.go:28] interesting pod/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 10 22:58:16.858389 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:16.858104 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" podUID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 10 22:58:17.861962 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:17.861925 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:58:17.862345 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:17.862191 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:58:17.863033 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:58:17.863012 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 22:59:03.915048 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:03.915009 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:59:36.428456 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:36.428419 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:59:37.386026 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:37.385994 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:59:37.486266 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:37.486233 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc"] Jul 10 22:59:37.486710 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:37.486492 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" podUID="b104947a-8899-4694-906d-b531b0a06e4b" containerName="maas-api" containerID="cri-o://5dc0f6c15fe5c6fb5d9f8e77caff8d14e834d6554b84cba1d0b2cba850f93faa" gracePeriod=30 Jul 10 22:59:37.516612 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:37.516582 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:59:37.517049 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:37.517013 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" podUID="b104947a-8899-4694-906d-b531b0a06e4b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.59:8443/health\": dial tcp 10.134.0.59:8443: connect: connection refused" Jul 10 22:59:37.617240 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:37.617207 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:59:37.918732 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:37.918700 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 22:59:37.944129 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:37.944107 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:59:38.097508 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.097367 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b104947a-8899-4694-906d-b531b0a06e4b-openshift-service-ca\") pod \"b104947a-8899-4694-906d-b531b0a06e4b\" (UID: \"b104947a-8899-4694-906d-b531b0a06e4b\") " Jul 10 22:59:38.097508 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.097431 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b104947a-8899-4694-906d-b531b0a06e4b-maas-api-tls\") pod \"b104947a-8899-4694-906d-b531b0a06e4b\" (UID: \"b104947a-8899-4694-906d-b531b0a06e4b\") " Jul 10 22:59:38.097508 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.097474 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vwzw\" (UniqueName: \"kubernetes.io/projected/b104947a-8899-4694-906d-b531b0a06e4b-kube-api-access-2vwzw\") pod \"b104947a-8899-4694-906d-b531b0a06e4b\" (UID: \"b104947a-8899-4694-906d-b531b0a06e4b\") " Jul 10 22:59:38.097877 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.097855 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b104947a-8899-4694-906d-b531b0a06e4b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b104947a-8899-4694-906d-b531b0a06e4b" (UID: "b104947a-8899-4694-906d-b531b0a06e4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:59:38.099634 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.099606 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b104947a-8899-4694-906d-b531b0a06e4b-kube-api-access-2vwzw" (OuterVolumeSpecName: "kube-api-access-2vwzw") pod "b104947a-8899-4694-906d-b531b0a06e4b" (UID: "b104947a-8899-4694-906d-b531b0a06e4b"). InnerVolumeSpecName "kube-api-access-2vwzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:59:38.099741 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.099649 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b104947a-8899-4694-906d-b531b0a06e4b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b104947a-8899-4694-906d-b531b0a06e4b" (UID: "b104947a-8899-4694-906d-b531b0a06e4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:59:38.111843 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.111812 2565 generic.go:356] "Generic (PLEG): container finished" podID="b104947a-8899-4694-906d-b531b0a06e4b" containerID="5dc0f6c15fe5c6fb5d9f8e77caff8d14e834d6554b84cba1d0b2cba850f93faa" exitCode=0 Jul 10 22:59:38.111973 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.111863 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" Jul 10 22:59:38.111973 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.111902 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" event={"ID":"b104947a-8899-4694-906d-b531b0a06e4b","Type":"ContainerDied","Data":"5dc0f6c15fe5c6fb5d9f8e77caff8d14e834d6554b84cba1d0b2cba850f93faa"} Jul 10 22:59:38.111973 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.111959 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc" event={"ID":"b104947a-8899-4694-906d-b531b0a06e4b","Type":"ContainerDied","Data":"38127207c9e65798caa9e8d953265a9fb857858d8955213c92a8cf5789247e29"} Jul 10 22:59:38.112081 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.111982 2565 scope.go:117] "RemoveContainer" containerID="5dc0f6c15fe5c6fb5d9f8e77caff8d14e834d6554b84cba1d0b2cba850f93faa" Jul 10 22:59:38.121212 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.121193 2565 scope.go:117] "RemoveContainer" containerID="5dc0f6c15fe5c6fb5d9f8e77caff8d14e834d6554b84cba1d0b2cba850f93faa" Jul 10 22:59:38.121485 ip-10-0-136-68 kubenswrapper[2565]: E0710 22:59:38.121467 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc0f6c15fe5c6fb5d9f8e77caff8d14e834d6554b84cba1d0b2cba850f93faa\": container with ID starting with 5dc0f6c15fe5c6fb5d9f8e77caff8d14e834d6554b84cba1d0b2cba850f93faa not found: ID does not exist" containerID="5dc0f6c15fe5c6fb5d9f8e77caff8d14e834d6554b84cba1d0b2cba850f93faa" Jul 10 22:59:38.121530 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.121495 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc0f6c15fe5c6fb5d9f8e77caff8d14e834d6554b84cba1d0b2cba850f93faa"} err="failed to get container status \"5dc0f6c15fe5c6fb5d9f8e77caff8d14e834d6554b84cba1d0b2cba850f93faa\": rpc error: code = NotFound desc = could not find container \"5dc0f6c15fe5c6fb5d9f8e77caff8d14e834d6554b84cba1d0b2cba850f93faa\": container with ID starting with 5dc0f6c15fe5c6fb5d9f8e77caff8d14e834d6554b84cba1d0b2cba850f93faa not found: ID does not exist" Jul 10 22:59:38.132351 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.132317 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc"] Jul 10 22:59:38.136112 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.136088 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8c45c-f79485c5c-kd4xc"] Jul 10 22:59:38.198859 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.198826 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b104947a-8899-4694-906d-b531b0a06e4b-openshift-service-ca\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:59:38.198859 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.198855 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b104947a-8899-4694-906d-b531b0a06e4b-maas-api-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:59:38.198859 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:38.198866 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vwzw\" (UniqueName: \"kubernetes.io/projected/b104947a-8899-4694-906d-b531b0a06e4b-kube-api-access-2vwzw\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 22:59:39.691297 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:39.691262 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b104947a-8899-4694-906d-b531b0a06e4b" path="/var/lib/kubelet/pods/b104947a-8899-4694-906d-b531b0a06e4b/volumes" Jul 10 22:59:42.997068 ip-10-0-136-68 kubenswrapper[2565]: I0710 22:59:42.997034 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:19.978760 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:19.978727 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:20.315033 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:20.314948 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs"] Jul 10 23:00:20.315181 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:20.315154 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" podUID="f6f25824-9f93-4530-9bb6-8636686a1971" containerName="istio-proxy" containerID="cri-o://cb07b7178f571aaa99eb5461d9b8d7d6b0226c13d55967aba8997486e0deb83a" gracePeriod=30 Jul 10 23:00:20.567662 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:20.567566 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:20.622817 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:20.622767 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp"] Jul 10 23:00:20.623081 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:20.623052 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" podUID="97059587-6725-432f-b7bf-a65226e08ecd" containerName="maas-api" containerID="cri-o://d032d401c8a467c1fba0836ca4b33447e6806cbbae23646fc3cfa2d573b94b21" gracePeriod=30 Jul 10 23:00:20.648939 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:20.648889 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:20.711769 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:20.711736 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:21.070749 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.070723 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 23:00:21.084313 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.084280 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:21.197063 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.196962 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97059587-6725-432f-b7bf-a65226e08ecd-openshift-service-ca\") pod \"97059587-6725-432f-b7bf-a65226e08ecd\" (UID: \"97059587-6725-432f-b7bf-a65226e08ecd\") " Jul 10 23:00:21.197063 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.197034 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mkfn\" (UniqueName: \"kubernetes.io/projected/97059587-6725-432f-b7bf-a65226e08ecd-kube-api-access-2mkfn\") pod \"97059587-6725-432f-b7bf-a65226e08ecd\" (UID: \"97059587-6725-432f-b7bf-a65226e08ecd\") " Jul 10 23:00:21.197308 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.197071 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97059587-6725-432f-b7bf-a65226e08ecd-maas-api-tls\") pod \"97059587-6725-432f-b7bf-a65226e08ecd\" (UID: \"97059587-6725-432f-b7bf-a65226e08ecd\") " Jul 10 23:00:21.197375 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.197353 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97059587-6725-432f-b7bf-a65226e08ecd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97059587-6725-432f-b7bf-a65226e08ecd" (UID: "97059587-6725-432f-b7bf-a65226e08ecd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:00:21.199257 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.199229 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97059587-6725-432f-b7bf-a65226e08ecd-kube-api-access-2mkfn" (OuterVolumeSpecName: "kube-api-access-2mkfn") pod "97059587-6725-432f-b7bf-a65226e08ecd" (UID: "97059587-6725-432f-b7bf-a65226e08ecd"). InnerVolumeSpecName "kube-api-access-2mkfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:00:21.199257 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.199239 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97059587-6725-432f-b7bf-a65226e08ecd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97059587-6725-432f-b7bf-a65226e08ecd" (UID: "97059587-6725-432f-b7bf-a65226e08ecd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 23:00:21.298147 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.298106 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97059587-6725-432f-b7bf-a65226e08ecd-openshift-service-ca\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:21.298147 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.298140 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mkfn\" (UniqueName: \"kubernetes.io/projected/97059587-6725-432f-b7bf-a65226e08ecd-kube-api-access-2mkfn\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:21.298147 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.298152 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97059587-6725-432f-b7bf-a65226e08ecd-maas-api-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:21.339620 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.339584 2565 generic.go:356] "Generic (PLEG): container finished" podID="97059587-6725-432f-b7bf-a65226e08ecd" containerID="d032d401c8a467c1fba0836ca4b33447e6806cbbae23646fc3cfa2d573b94b21" exitCode=0 Jul 10 23:00:21.339854 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.339640 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" Jul 10 23:00:21.339854 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.339662 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" event={"ID":"97059587-6725-432f-b7bf-a65226e08ecd","Type":"ContainerDied","Data":"d032d401c8a467c1fba0836ca4b33447e6806cbbae23646fc3cfa2d573b94b21"} Jul 10 23:00:21.339854 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.339719 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp" event={"ID":"97059587-6725-432f-b7bf-a65226e08ecd","Type":"ContainerDied","Data":"2faebce05cfde8d99c337acdeed325635d81c16530dc330e06c80907dba42e57"} Jul 10 23:00:21.339854 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.339742 2565 scope.go:117] "RemoveContainer" containerID="d032d401c8a467c1fba0836ca4b33447e6806cbbae23646fc3cfa2d573b94b21" Jul 10 23:00:21.349354 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.349331 2565 scope.go:117] "RemoveContainer" containerID="d032d401c8a467c1fba0836ca4b33447e6806cbbae23646fc3cfa2d573b94b21" Jul 10 23:00:21.349616 ip-10-0-136-68 kubenswrapper[2565]: E0710 23:00:21.349598 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d032d401c8a467c1fba0836ca4b33447e6806cbbae23646fc3cfa2d573b94b21\": container with ID starting with d032d401c8a467c1fba0836ca4b33447e6806cbbae23646fc3cfa2d573b94b21 not found: ID does not exist" containerID="d032d401c8a467c1fba0836ca4b33447e6806cbbae23646fc3cfa2d573b94b21" Jul 10 23:00:21.349662 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.349625 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d032d401c8a467c1fba0836ca4b33447e6806cbbae23646fc3cfa2d573b94b21"} err="failed to get container status \"d032d401c8a467c1fba0836ca4b33447e6806cbbae23646fc3cfa2d573b94b21\": rpc error: code = NotFound desc = could not find container \"d032d401c8a467c1fba0836ca4b33447e6806cbbae23646fc3cfa2d573b94b21\": container with ID starting with d032d401c8a467c1fba0836ca4b33447e6806cbbae23646fc3cfa2d573b94b21 not found: ID does not exist" Jul 10 23:00:21.361483 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.361457 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp"] Jul 10 23:00:21.363605 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.363585 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c49b05-88b7d559c-jvfvp"] Jul 10 23:00:21.691654 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:21.691624 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97059587-6725-432f-b7bf-a65226e08ecd" path="/var/lib/kubelet/pods/97059587-6725-432f-b7bf-a65226e08ecd/volumes" Jul 10 23:00:25.574012 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.573950 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 23:00:25.738739 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.738689 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-credential-socket\") pod \"f6f25824-9f93-4530-9bb6-8636686a1971\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " Jul 10 23:00:25.738739 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.738752 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-istio-envoy\") pod \"f6f25824-9f93-4530-9bb6-8636686a1971\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " Jul 10 23:00:25.739030 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.738776 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f25824-9f93-4530-9bb6-8636686a1971-istiod-ca-cert\") pod \"f6f25824-9f93-4530-9bb6-8636686a1971\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " Jul 10 23:00:25.739030 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.738834 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-workload-certs\") pod \"f6f25824-9f93-4530-9bb6-8636686a1971\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " Jul 10 23:00:25.739030 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.738968 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f25824-9f93-4530-9bb6-8636686a1971-istio-podinfo\") pod \"f6f25824-9f93-4530-9bb6-8636686a1971\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " Jul 10 23:00:25.739030 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.739005 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-workload-socket\") pod \"f6f25824-9f93-4530-9bb6-8636686a1971\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " Jul 10 23:00:25.739249 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.739041 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd9n6\" (UniqueName: \"kubernetes.io/projected/f6f25824-9f93-4530-9bb6-8636686a1971-kube-api-access-dd9n6\") pod \"f6f25824-9f93-4530-9bb6-8636686a1971\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " Jul 10 23:00:25.739249 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.739069 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f25824-9f93-4530-9bb6-8636686a1971-istio-token\") pod \"f6f25824-9f93-4530-9bb6-8636686a1971\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " Jul 10 23:00:25.739249 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.739093 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-istio-data\") pod \"f6f25824-9f93-4530-9bb6-8636686a1971\" (UID: \"f6f25824-9f93-4530-9bb6-8636686a1971\") " Jul 10 23:00:25.739249 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.739064 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f6f25824-9f93-4530-9bb6-8636686a1971" (UID: "f6f25824-9f93-4530-9bb6-8636686a1971"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:00:25.739249 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.739139 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f6f25824-9f93-4530-9bb6-8636686a1971" (UID: "f6f25824-9f93-4530-9bb6-8636686a1971"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:00:25.739526 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.739277 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f6f25824-9f93-4530-9bb6-8636686a1971" (UID: "f6f25824-9f93-4530-9bb6-8636686a1971"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:00:25.739526 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.739507 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-workload-certs\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:25.739637 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.739528 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-workload-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:25.739637 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.739542 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-credential-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:25.740122 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.740089 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-istio-data" (OuterVolumeSpecName: "istio-data") pod "f6f25824-9f93-4530-9bb6-8636686a1971" (UID: "f6f25824-9f93-4530-9bb6-8636686a1971"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:00:25.740229 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.740180 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f25824-9f93-4530-9bb6-8636686a1971-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f6f25824-9f93-4530-9bb6-8636686a1971" (UID: "f6f25824-9f93-4530-9bb6-8636686a1971"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:00:25.741172 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.741146 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6f25824-9f93-4530-9bb6-8636686a1971-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f6f25824-9f93-4530-9bb6-8636686a1971" (UID: "f6f25824-9f93-4530-9bb6-8636686a1971"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 23:00:25.741255 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.741170 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f6f25824-9f93-4530-9bb6-8636686a1971" (UID: "f6f25824-9f93-4530-9bb6-8636686a1971"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:00:25.741255 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.741206 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f25824-9f93-4530-9bb6-8636686a1971-kube-api-access-dd9n6" (OuterVolumeSpecName: "kube-api-access-dd9n6") pod "f6f25824-9f93-4530-9bb6-8636686a1971" (UID: "f6f25824-9f93-4530-9bb6-8636686a1971"). InnerVolumeSpecName "kube-api-access-dd9n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:00:25.741330 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.741312 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f25824-9f93-4530-9bb6-8636686a1971-istio-token" (OuterVolumeSpecName: "istio-token") pod "f6f25824-9f93-4530-9bb6-8636686a1971" (UID: "f6f25824-9f93-4530-9bb6-8636686a1971"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:00:25.840223 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.840122 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f25824-9f93-4530-9bb6-8636686a1971-istio-podinfo\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:25.840223 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.840165 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd9n6\" (UniqueName: \"kubernetes.io/projected/f6f25824-9f93-4530-9bb6-8636686a1971-kube-api-access-dd9n6\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:25.840223 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.840175 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f25824-9f93-4530-9bb6-8636686a1971-istio-token\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:25.840223 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.840184 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-istio-data\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:25.840223 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.840191 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f25824-9f93-4530-9bb6-8636686a1971-istio-envoy\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:25.840223 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:25.840198 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f25824-9f93-4530-9bb6-8636686a1971-istiod-ca-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:26.369125 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.369085 2565 generic.go:356] "Generic (PLEG): container finished" podID="f6f25824-9f93-4530-9bb6-8636686a1971" containerID="cb07b7178f571aaa99eb5461d9b8d7d6b0226c13d55967aba8997486e0deb83a" exitCode=0 Jul 10 23:00:26.369313 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.369144 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" Jul 10 23:00:26.369313 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.369164 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" event={"ID":"f6f25824-9f93-4530-9bb6-8636686a1971","Type":"ContainerDied","Data":"cb07b7178f571aaa99eb5461d9b8d7d6b0226c13d55967aba8997486e0deb83a"} Jul 10 23:00:26.369313 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.369217 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs" event={"ID":"f6f25824-9f93-4530-9bb6-8636686a1971","Type":"ContainerDied","Data":"0404adbc14404735f0823c3ef16b6fc4d5add9352e0ddba361ef85ec5ad305a5"} Jul 10 23:00:26.369313 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.369241 2565 scope.go:117] "RemoveContainer" containerID="cb07b7178f571aaa99eb5461d9b8d7d6b0226c13d55967aba8997486e0deb83a" Jul 10 23:00:26.378892 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.378871 2565 scope.go:117] "RemoveContainer" containerID="efbe9355eb2ec742c6bf3daa8f868d9ae51503f5398a12dd6af3bece9a5f0456" Jul 10 23:00:26.387710 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.387661 2565 scope.go:117] "RemoveContainer" containerID="cb07b7178f571aaa99eb5461d9b8d7d6b0226c13d55967aba8997486e0deb83a" Jul 10 23:00:26.388160 ip-10-0-136-68 kubenswrapper[2565]: E0710 23:00:26.388036 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb07b7178f571aaa99eb5461d9b8d7d6b0226c13d55967aba8997486e0deb83a\": container with ID starting with cb07b7178f571aaa99eb5461d9b8d7d6b0226c13d55967aba8997486e0deb83a not found: ID does not exist" containerID="cb07b7178f571aaa99eb5461d9b8d7d6b0226c13d55967aba8997486e0deb83a" Jul 10 23:00:26.388160 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.388114 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb07b7178f571aaa99eb5461d9b8d7d6b0226c13d55967aba8997486e0deb83a"} err="failed to get container status \"cb07b7178f571aaa99eb5461d9b8d7d6b0226c13d55967aba8997486e0deb83a\": rpc error: code = NotFound desc = could not find container \"cb07b7178f571aaa99eb5461d9b8d7d6b0226c13d55967aba8997486e0deb83a\": container with ID starting with cb07b7178f571aaa99eb5461d9b8d7d6b0226c13d55967aba8997486e0deb83a not found: ID does not exist" Jul 10 23:00:26.388160 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.388140 2565 scope.go:117] "RemoveContainer" containerID="efbe9355eb2ec742c6bf3daa8f868d9ae51503f5398a12dd6af3bece9a5f0456" Jul 10 23:00:26.388550 ip-10-0-136-68 kubenswrapper[2565]: E0710 23:00:26.388530 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbe9355eb2ec742c6bf3daa8f868d9ae51503f5398a12dd6af3bece9a5f0456\": container with ID starting with efbe9355eb2ec742c6bf3daa8f868d9ae51503f5398a12dd6af3bece9a5f0456 not found: ID does not exist" containerID="efbe9355eb2ec742c6bf3daa8f868d9ae51503f5398a12dd6af3bece9a5f0456" Jul 10 23:00:26.388609 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.388556 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbe9355eb2ec742c6bf3daa8f868d9ae51503f5398a12dd6af3bece9a5f0456"} err="failed to get container status \"efbe9355eb2ec742c6bf3daa8f868d9ae51503f5398a12dd6af3bece9a5f0456\": rpc error: code = NotFound desc = could not find container \"efbe9355eb2ec742c6bf3daa8f868d9ae51503f5398a12dd6af3bece9a5f0456\": container with ID starting with efbe9355eb2ec742c6bf3daa8f868d9ae51503f5398a12dd6af3bece9a5f0456 not found: ID does not exist" Jul 10 23:00:26.390678 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.390658 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs"] Jul 10 23:00:26.394920 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.394898 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c8c45c-openshift-default-77b4956f49-6xxjs"] Jul 10 23:00:26.764501 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.764466 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:26.870160 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:26.870120 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:27.022495 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.022383 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm"] Jul 10 23:00:27.022698 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.022669 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" podUID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerName="istio-proxy" containerID="cri-o://40dacb91e8156f3aaabed395bbcb79503367e974a3203dd04d85767094ae0e6d" gracePeriod=30 Jul 10 23:00:27.068212 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.068178 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:27.278179 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.278088 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:27.625482 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.625377 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:27.690703 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.690669 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6f25824-9f93-4530-9bb6-8636686a1971" path="/var/lib/kubelet/pods/f6f25824-9f93-4530-9bb6-8636686a1971/volumes" Jul 10 23:00:27.914628 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.914552 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv"] Jul 10 23:00:27.915033 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.914988 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97059587-6725-432f-b7bf-a65226e08ecd" containerName="maas-api" Jul 10 23:00:27.915033 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.915001 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97059587-6725-432f-b7bf-a65226e08ecd" containerName="maas-api" Jul 10 23:00:27.915033 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.915016 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b104947a-8899-4694-906d-b531b0a06e4b" containerName="maas-api" Jul 10 23:00:27.915033 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.915021 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b104947a-8899-4694-906d-b531b0a06e4b" containerName="maas-api" Jul 10 23:00:27.915033 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.915027 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6f25824-9f93-4530-9bb6-8636686a1971" containerName="istio-proxy" Jul 10 23:00:27.915033 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.915033 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6f25824-9f93-4530-9bb6-8636686a1971" containerName="istio-proxy" Jul 10 23:00:27.915220 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.915045 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6f25824-9f93-4530-9bb6-8636686a1971" containerName="istio-proxy" Jul 10 23:00:27.915220 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.915051 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6f25824-9f93-4530-9bb6-8636686a1971" containerName="istio-proxy" Jul 10 23:00:27.915220 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.915108 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97059587-6725-432f-b7bf-a65226e08ecd" containerName="maas-api" Jul 10 23:00:27.915220 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.915116 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6f25824-9f93-4530-9bb6-8636686a1971" containerName="istio-proxy" Jul 10 23:00:27.915220 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.915121 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6f25824-9f93-4530-9bb6-8636686a1971" containerName="istio-proxy" Jul 10 23:00:27.915220 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.915128 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b104947a-8899-4694-906d-b531b0a06e4b" containerName="maas-api" Jul 10 23:00:27.919761 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.919738 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:27.921996 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.921975 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-54ec41-openshift-default-dockercfg-xlv6v\"" Jul 10 23:00:27.929594 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.929570 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv"] Jul 10 23:00:27.938867 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.938828 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:27.962291 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.962259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-credential-socket\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:27.962437 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.962302 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-workload-certs\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:27.962437 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.962385 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-workload-socket\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:27.962437 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.962425 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f8f118-ea7a-42c7-9577-24100c53d74a-istiod-ca-cert\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:27.962554 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.962464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-data\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:27.962554 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.962486 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-envoy\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:27.962554 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.962503 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-podinfo\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:27.962554 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.962522 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-token\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:27.962677 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:27.962612 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkfgv\" (UniqueName: \"kubernetes.io/projected/80f8f118-ea7a-42c7-9577-24100c53d74a-kube-api-access-nkfgv\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.063040 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.063008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-token\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.063234 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.063160 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkfgv\" (UniqueName: \"kubernetes.io/projected/80f8f118-ea7a-42c7-9577-24100c53d74a-kube-api-access-nkfgv\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.063234 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.063192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-credential-socket\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.063332 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.063315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-workload-certs\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.063409 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.063390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-workload-socket\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.063470 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.063437 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f8f118-ea7a-42c7-9577-24100c53d74a-istiod-ca-cert\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.063530 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.063470 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-data\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.063530 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.063505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-envoy\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.063642 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.063532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-podinfo\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.063696 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.063649 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-credential-socket\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.063696 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.063666 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-workload-certs\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.063863 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.063840 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-data\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.064073 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.064040 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-workload-socket\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.064215 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.064181 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f8f118-ea7a-42c7-9577-24100c53d74a-istiod-ca-cert\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.065735 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.065711 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-envoy\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.065975 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.065958 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-podinfo\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.081488 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.081462 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkfgv\" (UniqueName: \"kubernetes.io/projected/80f8f118-ea7a-42c7-9577-24100c53d74a-kube-api-access-nkfgv\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.094353 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.094320 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-token\") pod \"e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.100671 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.100638 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:28.231952 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.231917 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:28.383962 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.383936 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv"] Jul 10 23:00:28.385851 ip-10-0-136-68 kubenswrapper[2565]: W0710 23:00:28.385822 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f8f118_ea7a_42c7_9577_24100c53d74a.slice/crio-d0adbf7f53bdc6d12bf7fffa48afbefb65944accd7509907862fc9f2074ad099 WatchSource:0}: Error finding container d0adbf7f53bdc6d12bf7fffa48afbefb65944accd7509907862fc9f2074ad099: Status 404 returned error can't find the container with id d0adbf7f53bdc6d12bf7fffa48afbefb65944accd7509907862fc9f2074ad099 Jul 10 23:00:28.387922 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.387891 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 23:00:28.387984 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.387959 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 23:00:28.388026 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:28.387988 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 23:00:29.388747 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:29.388703 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" event={"ID":"80f8f118-ea7a-42c7-9577-24100c53d74a","Type":"ContainerStarted","Data":"b811e5da9d1a0421ec73ed28ac7fab39e1cb19c4c14bcfe5235d96de035355ab"} Jul 10 23:00:29.389252 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:29.388815 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" event={"ID":"80f8f118-ea7a-42c7-9577-24100c53d74a","Type":"ContainerStarted","Data":"d0adbf7f53bdc6d12bf7fffa48afbefb65944accd7509907862fc9f2074ad099"} Jul 10 23:00:29.409942 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:29.409893 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" podStartSLOduration=2.4098783360000002 podStartE2EDuration="2.409878336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:00:29.408860911 +0000 UTC m=+2293.241745294" watchObservedRunningTime="2026-07-10 23:00:29.409878336 +0000 UTC m=+2293.242762719" Jul 10 23:00:30.232243 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:30.232202 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:30.237250 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:30.237228 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:30.394382 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:30.394353 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:30.395278 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:30.395259 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:00:31.344253 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:31.344215 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:31.442050 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:31.442015 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:00:32.289234 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.289172 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 23:00:32.303402 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.303378 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-workload-certs\") pod \"f79792ff-8752-42c6-b14b-3fc648ddd168\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " Jul 10 23:00:32.303513 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.303416 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-data\") pod \"f79792ff-8752-42c6-b14b-3fc648ddd168\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " Jul 10 23:00:32.303513 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.303433 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79792ff-8752-42c6-b14b-3fc648ddd168-istiod-ca-cert\") pod \"f79792ff-8752-42c6-b14b-3fc648ddd168\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " Jul 10 23:00:32.303513 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.303451 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-workload-socket\") pod \"f79792ff-8752-42c6-b14b-3fc648ddd168\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " Jul 10 23:00:32.303513 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.303489 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmwt8\" (UniqueName: \"kubernetes.io/projected/f79792ff-8752-42c6-b14b-3fc648ddd168-kube-api-access-wmwt8\") pod \"f79792ff-8752-42c6-b14b-3fc648ddd168\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " Jul 10 23:00:32.303704 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.303517 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-podinfo\") pod \"f79792ff-8752-42c6-b14b-3fc648ddd168\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " Jul 10 23:00:32.303704 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.303550 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-envoy\") pod \"f79792ff-8752-42c6-b14b-3fc648ddd168\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " Jul 10 23:00:32.303704 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.303575 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-credential-socket\") pod \"f79792ff-8752-42c6-b14b-3fc648ddd168\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " Jul 10 23:00:32.303704 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.303606 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-token\") pod \"f79792ff-8752-42c6-b14b-3fc648ddd168\" (UID: \"f79792ff-8752-42c6-b14b-3fc648ddd168\") " Jul 10 23:00:32.303704 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.303664 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f79792ff-8752-42c6-b14b-3fc648ddd168" (UID: "f79792ff-8752-42c6-b14b-3fc648ddd168"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:00:32.303974 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.303785 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f79792ff-8752-42c6-b14b-3fc648ddd168" (UID: "f79792ff-8752-42c6-b14b-3fc648ddd168"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:00:32.303974 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.303902 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f79792ff-8752-42c6-b14b-3fc648ddd168" (UID: "f79792ff-8752-42c6-b14b-3fc648ddd168"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:00:32.304094 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.303999 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-workload-certs\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:32.304094 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.304017 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-workload-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:32.304094 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.304031 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-credential-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:32.304294 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.304268 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-data" (OuterVolumeSpecName: "istio-data") pod "f79792ff-8752-42c6-b14b-3fc648ddd168" (UID: "f79792ff-8752-42c6-b14b-3fc648ddd168"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:00:32.304453 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.304429 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79792ff-8752-42c6-b14b-3fc648ddd168-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f79792ff-8752-42c6-b14b-3fc648ddd168" (UID: "f79792ff-8752-42c6-b14b-3fc648ddd168"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:00:32.305908 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.305860 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-token" (OuterVolumeSpecName: "istio-token") pod "f79792ff-8752-42c6-b14b-3fc648ddd168" (UID: "f79792ff-8752-42c6-b14b-3fc648ddd168"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:00:32.305908 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.305887 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79792ff-8752-42c6-b14b-3fc648ddd168-kube-api-access-wmwt8" (OuterVolumeSpecName: "kube-api-access-wmwt8") pod "f79792ff-8752-42c6-b14b-3fc648ddd168" (UID: "f79792ff-8752-42c6-b14b-3fc648ddd168"). InnerVolumeSpecName "kube-api-access-wmwt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:00:32.306089 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.305929 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f79792ff-8752-42c6-b14b-3fc648ddd168" (UID: "f79792ff-8752-42c6-b14b-3fc648ddd168"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 23:00:32.306089 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.306036 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f79792ff-8752-42c6-b14b-3fc648ddd168" (UID: "f79792ff-8752-42c6-b14b-3fc648ddd168"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:00:32.404452 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.404419 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-envoy\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:32.404452 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.404447 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-token\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:32.404452 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.404456 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-data\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:32.404698 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.404465 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79792ff-8752-42c6-b14b-3fc648ddd168-istiod-ca-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:32.404698 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.404474 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmwt8\" (UniqueName: \"kubernetes.io/projected/f79792ff-8752-42c6-b14b-3fc648ddd168-kube-api-access-wmwt8\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:32.404698 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.404483 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79792ff-8752-42c6-b14b-3fc648ddd168-istio-podinfo\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:00:32.405515 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.405493 2565 generic.go:356] "Generic (PLEG): container finished" podID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerID="40dacb91e8156f3aaabed395bbcb79503367e974a3203dd04d85767094ae0e6d" exitCode=0 Jul 10 23:00:32.405598 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.405544 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" Jul 10 23:00:32.405598 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.405580 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" event={"ID":"f79792ff-8752-42c6-b14b-3fc648ddd168","Type":"ContainerDied","Data":"40dacb91e8156f3aaabed395bbcb79503367e974a3203dd04d85767094ae0e6d"} Jul 10 23:00:32.405684 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.405634 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm" event={"ID":"f79792ff-8752-42c6-b14b-3fc648ddd168","Type":"ContainerDied","Data":"83351d703d3623621a75e9535cb26e58e83ed8ded6816905ba8dfc521a24f2eb"} Jul 10 23:00:32.405684 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.405656 2565 scope.go:117] "RemoveContainer" containerID="40dacb91e8156f3aaabed395bbcb79503367e974a3203dd04d85767094ae0e6d" Jul 10 23:00:32.415231 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.415207 2565 scope.go:117] "RemoveContainer" containerID="361600e0af086b0b4c40332e895aa4ae414b1614deca651751db01ca01548e63" Jul 10 23:00:32.423145 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.423119 2565 scope.go:117] "RemoveContainer" containerID="40dacb91e8156f3aaabed395bbcb79503367e974a3203dd04d85767094ae0e6d" Jul 10 23:00:32.423420 ip-10-0-136-68 kubenswrapper[2565]: E0710 23:00:32.423401 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40dacb91e8156f3aaabed395bbcb79503367e974a3203dd04d85767094ae0e6d\": container with ID starting with 40dacb91e8156f3aaabed395bbcb79503367e974a3203dd04d85767094ae0e6d not found: ID does not exist" containerID="40dacb91e8156f3aaabed395bbcb79503367e974a3203dd04d85767094ae0e6d" Jul 10 23:00:32.423487 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.423429 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40dacb91e8156f3aaabed395bbcb79503367e974a3203dd04d85767094ae0e6d"} err="failed to get container status \"40dacb91e8156f3aaabed395bbcb79503367e974a3203dd04d85767094ae0e6d\": rpc error: code = NotFound desc = could not find container \"40dacb91e8156f3aaabed395bbcb79503367e974a3203dd04d85767094ae0e6d\": container with ID starting with 40dacb91e8156f3aaabed395bbcb79503367e974a3203dd04d85767094ae0e6d not found: ID does not exist" Jul 10 23:00:32.423487 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.423447 2565 scope.go:117] "RemoveContainer" containerID="361600e0af086b0b4c40332e895aa4ae414b1614deca651751db01ca01548e63" Jul 10 23:00:32.423701 ip-10-0-136-68 kubenswrapper[2565]: E0710 23:00:32.423683 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361600e0af086b0b4c40332e895aa4ae414b1614deca651751db01ca01548e63\": container with ID starting with 361600e0af086b0b4c40332e895aa4ae414b1614deca651751db01ca01548e63 not found: ID does not exist" containerID="361600e0af086b0b4c40332e895aa4ae414b1614deca651751db01ca01548e63" Jul 10 23:00:32.423755 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.423707 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361600e0af086b0b4c40332e895aa4ae414b1614deca651751db01ca01548e63"} err="failed to get container status \"361600e0af086b0b4c40332e895aa4ae414b1614deca651751db01ca01548e63\": rpc error: code = NotFound desc = could not find container \"361600e0af086b0b4c40332e895aa4ae414b1614deca651751db01ca01548e63\": container with ID starting with 361600e0af086b0b4c40332e895aa4ae414b1614deca651751db01ca01548e63 not found: ID does not exist" Jul 10 23:00:32.427389 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.427369 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm"] Jul 10 23:00:32.431954 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:32.431928 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c49b05-openshift-default-fd468cc97-rclbm"] Jul 10 23:00:33.691244 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:33.691209 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f79792ff-8752-42c6-b14b-3fc648ddd168" path="/var/lib/kubelet/pods/f79792ff-8752-42c6-b14b-3fc648ddd168/volumes" Jul 10 23:00:35.868610 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:00:35.868571 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:01:00.721930 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:01:00.721893 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:01:11.766402 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:01:11.766367 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:01:53.505647 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:01:53.505599 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:02:18.434931 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:02:18.434902 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 23:02:18.441265 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:02:18.441244 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 23:02:33.877189 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:02:33.877147 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:03:05.747623 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:05.747541 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:03:05.796185 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:05.796154 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:03:40.818718 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:40.818680 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:03:41.152945 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:41.152854 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:03:47.011433 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:47.011390 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:03:47.092194 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:47.092155 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:03:47.270558 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:47.270471 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv"] Jul 10 23:03:47.270770 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:47.270746 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" podUID="80f8f118-ea7a-42c7-9577-24100c53d74a" containerName="istio-proxy" containerID="cri-o://b811e5da9d1a0421ec73ed28ac7fab39e1cb19c4c14bcfe5235d96de035355ab" gracePeriod=30 Jul 10 23:03:47.289322 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:47.289291 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:03:47.510772 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:47.510741 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:03:52.434135 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.434098 2565 generic.go:356] "Generic (PLEG): container finished" podID="80f8f118-ea7a-42c7-9577-24100c53d74a" containerID="b811e5da9d1a0421ec73ed28ac7fab39e1cb19c4c14bcfe5235d96de035355ab" exitCode=0 Jul 10 23:03:52.434496 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.434170 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" event={"ID":"80f8f118-ea7a-42c7-9577-24100c53d74a","Type":"ContainerDied","Data":"b811e5da9d1a0421ec73ed28ac7fab39e1cb19c4c14bcfe5235d96de035355ab"} Jul 10 23:03:52.517076 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.517048 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:03:52.565763 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.565676 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-token\") pod \"80f8f118-ea7a-42c7-9577-24100c53d74a\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " Jul 10 23:03:52.565763 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.565725 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-envoy\") pod \"80f8f118-ea7a-42c7-9577-24100c53d74a\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " Jul 10 23:03:52.565763 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.565753 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-workload-socket\") pod \"80f8f118-ea7a-42c7-9577-24100c53d74a\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " Jul 10 23:03:52.566067 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.565885 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-data\") pod \"80f8f118-ea7a-42c7-9577-24100c53d74a\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " Jul 10 23:03:52.566067 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.565937 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-credential-socket\") pod \"80f8f118-ea7a-42c7-9577-24100c53d74a\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " Jul 10 23:03:52.566067 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.566015 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f8f118-ea7a-42c7-9577-24100c53d74a-istiod-ca-cert\") pod \"80f8f118-ea7a-42c7-9577-24100c53d74a\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " Jul 10 23:03:52.566067 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.566047 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkfgv\" (UniqueName: \"kubernetes.io/projected/80f8f118-ea7a-42c7-9577-24100c53d74a-kube-api-access-nkfgv\") pod \"80f8f118-ea7a-42c7-9577-24100c53d74a\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " Jul 10 23:03:52.566279 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.566078 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-podinfo\") pod \"80f8f118-ea7a-42c7-9577-24100c53d74a\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " Jul 10 23:03:52.566279 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.566078 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "80f8f118-ea7a-42c7-9577-24100c53d74a" (UID: "80f8f118-ea7a-42c7-9577-24100c53d74a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:03:52.566279 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.566113 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-workload-certs\") pod \"80f8f118-ea7a-42c7-9577-24100c53d74a\" (UID: \"80f8f118-ea7a-42c7-9577-24100c53d74a\") " Jul 10 23:03:52.566279 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.566132 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-data" (OuterVolumeSpecName: "istio-data") pod "80f8f118-ea7a-42c7-9577-24100c53d74a" (UID: "80f8f118-ea7a-42c7-9577-24100c53d74a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:03:52.566279 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.566249 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "80f8f118-ea7a-42c7-9577-24100c53d74a" (UID: "80f8f118-ea7a-42c7-9577-24100c53d74a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:03:52.566553 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.566346 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f8f118-ea7a-42c7-9577-24100c53d74a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "80f8f118-ea7a-42c7-9577-24100c53d74a" (UID: "80f8f118-ea7a-42c7-9577-24100c53d74a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:03:52.566553 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.566486 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f8f118-ea7a-42c7-9577-24100c53d74a-istiod-ca-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:03:52.566553 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.566496 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "80f8f118-ea7a-42c7-9577-24100c53d74a" (UID: "80f8f118-ea7a-42c7-9577-24100c53d74a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:03:52.566553 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.566506 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-workload-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:03:52.566553 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.566535 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-data\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:03:52.566553 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.566552 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-credential-socket\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:03:52.568174 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.568151 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-token" (OuterVolumeSpecName: "istio-token") pod "80f8f118-ea7a-42c7-9577-24100c53d74a" (UID: "80f8f118-ea7a-42c7-9577-24100c53d74a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:03:52.568289 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.568245 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "80f8f118-ea7a-42c7-9577-24100c53d74a" (UID: "80f8f118-ea7a-42c7-9577-24100c53d74a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 23:03:52.568617 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.568593 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "80f8f118-ea7a-42c7-9577-24100c53d74a" (UID: "80f8f118-ea7a-42c7-9577-24100c53d74a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:03:52.568617 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.568605 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f8f118-ea7a-42c7-9577-24100c53d74a-kube-api-access-nkfgv" (OuterVolumeSpecName: "kube-api-access-nkfgv") pod "80f8f118-ea7a-42c7-9577-24100c53d74a" (UID: "80f8f118-ea7a-42c7-9577-24100c53d74a"). InnerVolumeSpecName "kube-api-access-nkfgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:03:52.667065 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.667024 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkfgv\" (UniqueName: \"kubernetes.io/projected/80f8f118-ea7a-42c7-9577-24100c53d74a-kube-api-access-nkfgv\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:03:52.667065 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.667056 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-podinfo\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:03:52.667065 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.667065 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-workload-certs\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:03:52.667065 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.667073 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-token\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:03:52.667065 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:52.667081 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f8f118-ea7a-42c7-9577-24100c53d74a-istio-envoy\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\"" Jul 10 23:03:53.440527 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:53.440499 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" Jul 10 23:03:53.441020 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:53.440538 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv" event={"ID":"80f8f118-ea7a-42c7-9577-24100c53d74a","Type":"ContainerDied","Data":"d0adbf7f53bdc6d12bf7fffa48afbefb65944accd7509907862fc9f2074ad099"} Jul 10 23:03:53.441020 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:53.440604 2565 scope.go:117] "RemoveContainer" containerID="b811e5da9d1a0421ec73ed28ac7fab39e1cb19c4c14bcfe5235d96de035355ab" Jul 10 23:03:53.462245 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:53.462212 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv"] Jul 10 23:03:53.468074 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:53.468050 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-54ec41-openshift-default-778dc9d847-m5kjv"] Jul 10 23:03:53.691125 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:53.691043 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f8f118-ea7a-42c7-9577-24100c53d74a" path="/var/lib/kubelet/pods/80f8f118-ea7a-42c7-9577-24100c53d74a/volumes" Jul 10 23:03:53.899220 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:53.899189 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bj7fm_30f4ad53-fa0f-4a7b-af87-20b48ae7783e/manager/0.log" Jul 10 23:03:54.146583 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:54.146551 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-795cbfd498-wtkgt_ba17e396-b47f-494b-9efd-057aa65a379a/manager/0.log" Jul 10 23:03:54.272430 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:54.272402 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zvqhz_20e39149-1c1b-44d5-939d-88ab9382fec7/server/0.log" Jul 10 23:03:54.401274 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:54.401191 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pgqgb_8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b/manager/0.log" Jul 10 23:03:54.797467 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:54.797440 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58f8cd5b6f-t88g8_ecd095a4-0f72-40d2-98c1-1e6f7421a3ea/copy-manifests/0.log" Jul 10 23:03:54.804330 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:54.804302 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58f8cd5b6f-t88g8_ecd095a4-0f72-40d2-98c1-1e6f7421a3ea/manager/0.log" Jul 10 23:03:55.153544 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:55.153469 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-65ccc6b695-xrkrg_3dd1c61b-9a82-48b4-bab5-b6c015301d93/maas-api/0.log" Jul 10 23:03:55.264081 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:55.264049 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-bzpbs_124b9f94-653b-441f-a4f9-2d8153c39cab/postgres/0.log" Jul 10 23:03:56.084900 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.084865 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w_22aca153-5823-443b-bd4e-c0009121fed9/util/0.log" Jul 10 23:03:56.093208 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.093181 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w_22aca153-5823-443b-bd4e-c0009121fed9/pull/0.log" Jul 10 23:03:56.099369 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.099349 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w_22aca153-5823-443b-bd4e-c0009121fed9/extract/0.log" Jul 10 23:03:56.211060 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.211025 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2_d4c96808-c75e-4edc-9e57-a2aae9c5c0d7/util/0.log" Jul 10 23:03:56.217526 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.217499 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2_d4c96808-c75e-4edc-9e57-a2aae9c5c0d7/pull/0.log" Jul 10 23:03:56.223826 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.223779 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2_d4c96808-c75e-4edc-9e57-a2aae9c5c0d7/extract/0.log" Jul 10 23:03:56.333005 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.332973 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl_e1accdaf-e3bd-4260-a715-737e35a214c1/util/0.log" Jul 10 23:03:56.338442 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.338383 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl_e1accdaf-e3bd-4260-a715-737e35a214c1/pull/0.log" Jul 10 23:03:56.343937 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.343920 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl_e1accdaf-e3bd-4260-a715-737e35a214c1/extract/0.log" Jul 10 23:03:56.452733 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.452699 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd_73e7fae0-1ba2-41e4-a679-85df4e8af497/util/0.log" Jul 10 23:03:56.462213 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.462183 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd_73e7fae0-1ba2-41e4-a679-85df4e8af497/pull/0.log" Jul 10 23:03:56.469403 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.469370 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd_73e7fae0-1ba2-41e4-a679-85df4e8af497/extract/0.log" Jul 10 23:03:56.719864 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.719760 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2hpnt_f7e899dc-a39f-418e-8ebe-e6fbb9967c09/manager/0.log" Jul 10 23:03:56.836105 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.836070 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-d9pwm_ad0186f0-e17b-4677-85a6-feb5853a3587/manager/0.log" Jul 10 23:03:56.945522 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:56.945494 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zxzzm_b6436dd9-f13f-4e4b-abf0-9fa23884e7ea/kuadrant-console-plugin/0.log" Jul 10 23:03:57.058333 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:57.058303 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zrvs8_99836a7c-8438-49b1-80f6-a34a30e2aa2b/registry-server/0.log" Jul 10 23:03:57.444489 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:57.444405 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-pzp7b_2ffca01f-3f3a-496c-93ac-0d46d9f27915/manager/0.log" Jul 10 23:03:57.797526 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:57.797493 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678spvxj_897507dc-df36-48fc-a691-a9e28fec3394/istio-proxy/0.log" Jul 10 23:03:58.040947 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:58.040918 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fc969b7d8-ps4x6_27a335eb-0ac8-49e3-8ea1-31491736893b/kube-auth-proxy/0.log" Jul 10 23:03:58.268958 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:58.268927 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sgnbx_a688576f-a240-451c-84a0-31ef6c45f6a9/istio-proxy/1.log" Jul 10 23:03:58.387117 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:58.387080 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-xqdf8_3ff97a1d-ad63-41f6-8768-b90dcfb72795/payload-pre-processing/0.log" Jul 10 23:03:58.515096 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:58.515064 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-h7xq9_0f9a29f3-8817-4929-bc2c-7d8be1c699b9/payload-processing/0.log" Jul 10 23:03:58.969761 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:03:58.969732 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-86cp5_493d0dc5-3aa2-455a-b4b7-efb46438745b/main/0.log" Jul 10 23:04:03.251021 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.250990 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qk7b9/must-gather-z6vld"] Jul 10 23:04:03.251414 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.251345 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerName="istio-proxy" Jul 10 23:04:03.251414 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.251358 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerName="istio-proxy" Jul 10 23:04:03.251414 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.251382 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f8f118-ea7a-42c7-9577-24100c53d74a" containerName="istio-proxy" Jul 10 23:04:03.251414 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.251388 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f8f118-ea7a-42c7-9577-24100c53d74a" containerName="istio-proxy" Jul 10 23:04:03.251608 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.251498 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f8f118-ea7a-42c7-9577-24100c53d74a" containerName="istio-proxy" Jul 10 23:04:03.251608 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.251513 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerName="istio-proxy" Jul 10 23:04:03.251608 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.251525 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerName="istio-proxy" Jul 10 23:04:03.251782 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.251619 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerName="istio-proxy" Jul 10 23:04:03.251782 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.251629 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f79792ff-8752-42c6-b14b-3fc648ddd168" containerName="istio-proxy" Jul 10 23:04:03.256192 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.256172 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qk7b9/must-gather-z6vld" Jul 10 23:04:03.259198 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.259120 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qk7b9\"/\"kube-root-ca.crt\"" Jul 10 23:04:03.264159 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.259536 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qk7b9\"/\"openshift-service-ca.crt\"" Jul 10 23:04:03.264159 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.259543 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qk7b9\"/\"default-dockercfg-n2xp8\"" Jul 10 23:04:03.268156 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.267129 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qk7b9/must-gather-z6vld"] Jul 10 23:04:03.268156 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.267847 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:04:03.349727 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.349691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/81802261-422c-407a-94ad-568f55ebb6d1-must-gather-output\") pod \"must-gather-z6vld\" (UID: \"81802261-422c-407a-94ad-568f55ebb6d1\") " pod="openshift-must-gather-qk7b9/must-gather-z6vld" Jul 10 23:04:03.349939 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.349875 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gx5z\" (UniqueName: \"kubernetes.io/projected/81802261-422c-407a-94ad-568f55ebb6d1-kube-api-access-6gx5z\") pod \"must-gather-z6vld\" (UID: \"81802261-422c-407a-94ad-568f55ebb6d1\") " pod="openshift-must-gather-qk7b9/must-gather-z6vld" Jul 10 23:04:03.367716 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.367687 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" limit=1024 Jul 10 23:04:03.450267 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.450236 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gx5z\" (UniqueName: \"kubernetes.io/projected/81802261-422c-407a-94ad-568f55ebb6d1-kube-api-access-6gx5z\") pod \"must-gather-z6vld\" (UID: \"81802261-422c-407a-94ad-568f55ebb6d1\") " pod="openshift-must-gather-qk7b9/must-gather-z6vld" Jul 10 23:04:03.450446 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.450284 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/81802261-422c-407a-94ad-568f55ebb6d1-must-gather-output\") pod \"must-gather-z6vld\" (UID: \"81802261-422c-407a-94ad-568f55ebb6d1\") " pod="openshift-must-gather-qk7b9/must-gather-z6vld" Jul 10 23:04:03.450647 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.450627 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/81802261-422c-407a-94ad-568f55ebb6d1-must-gather-output\") pod \"must-gather-z6vld\" (UID: \"81802261-422c-407a-94ad-568f55ebb6d1\") " pod="openshift-must-gather-qk7b9/must-gather-z6vld" Jul 10 23:04:03.458325 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.458291 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gx5z\" (UniqueName: \"kubernetes.io/projected/81802261-422c-407a-94ad-568f55ebb6d1-kube-api-access-6gx5z\") pod \"must-gather-z6vld\" (UID: \"81802261-422c-407a-94ad-568f55ebb6d1\") " pod="openshift-must-gather-qk7b9/must-gather-z6vld" Jul 10 23:04:03.570677 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.570577 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qk7b9/must-gather-z6vld" Jul 10 23:04:03.710015 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.709990 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qk7b9/must-gather-z6vld"] Jul 10 23:04:03.712091 ip-10-0-136-68 kubenswrapper[2565]: W0710 23:04:03.712064 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81802261_422c_407a_94ad_568f55ebb6d1.slice/crio-d87cad39fe7b98d98f38a1bf2c15ecf4eb0c8014db07e4e61a8a784b3401a6ee WatchSource:0}: Error finding container d87cad39fe7b98d98f38a1bf2c15ecf4eb0c8014db07e4e61a8a784b3401a6ee: Status 404 returned error can't find the container with id d87cad39fe7b98d98f38a1bf2c15ecf4eb0c8014db07e4e61a8a784b3401a6ee Jul 10 23:04:03.713908 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:03.713889 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 23:04:04.506065 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:04.506025 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qk7b9/must-gather-z6vld" event={"ID":"81802261-422c-407a-94ad-568f55ebb6d1","Type":"ContainerStarted","Data":"d87cad39fe7b98d98f38a1bf2c15ecf4eb0c8014db07e4e61a8a784b3401a6ee"} Jul 10 23:04:05.515966 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:05.515926 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qk7b9/must-gather-z6vld" event={"ID":"81802261-422c-407a-94ad-568f55ebb6d1","Type":"ContainerStarted","Data":"1694a403e5ac87e382844c813c024410712179b811a2da1b25b19677f87a8467"} Jul 10 23:04:05.515966 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:05.515971 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qk7b9/must-gather-z6vld" event={"ID":"81802261-422c-407a-94ad-568f55ebb6d1","Type":"ContainerStarted","Data":"11c42265b0554a8c72e3dfeaabed6b92ddba69f1c6b508f4f2155d86ac339faf"} Jul 10 23:04:05.536207 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:05.536139 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qk7b9/must-gather-z6vld" podStartSLOduration=1.7252472650000001 podStartE2EDuration="2.536115244s" totalImagesPullingTime="810.867979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:04:05.533974336 +0000 UTC m=+2509.366858719" watchObservedRunningTime="2026-07-10 23:04:05.536115244 +0000 UTC m=+2509.368999628" Jul 10 23:04:06.156908 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:06.156883 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b4thn_14b9b95f-6755-4116-bf6a-317e009798e2/global-pull-secret-syncer/0.log" Jul 10 23:04:06.299276 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:06.299246 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-95zkp_428ea365-0919-4562-bd39-f10dfdcda1d5/konnectivity-agent/0.log" Jul 10 23:04:06.405390 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:06.405363 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-68.ec2.internal_5a636bbdd5023b50004b217a7742580f/haproxy/0.log" Jul 10 23:04:10.219453 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.219368 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w_22aca153-5823-443b-bd4e-c0009121fed9/extract/0.log" Jul 10 23:04:10.254457 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.254423 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w_22aca153-5823-443b-bd4e-c0009121fed9/util/0.log" Jul 10 23:04:10.291993 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.291968 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598lc5w_22aca153-5823-443b-bd4e-c0009121fed9/pull/0.log" Jul 10 23:04:10.326679 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.326638 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2_d4c96808-c75e-4edc-9e57-a2aae9c5c0d7/extract/0.log" Jul 10 23:04:10.351147 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.351104 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2_d4c96808-c75e-4edc-9e57-a2aae9c5c0d7/util/0.log" Jul 10 23:04:10.414039 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.413979 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vh5t2_d4c96808-c75e-4edc-9e57-a2aae9c5c0d7/pull/0.log" Jul 10 23:04:10.465909 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.465856 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl_e1accdaf-e3bd-4260-a715-737e35a214c1/extract/0.log" Jul 10 23:04:10.489450 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.489419 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl_e1accdaf-e3bd-4260-a715-737e35a214c1/util/0.log" Jul 10 23:04:10.516559 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.516522 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwtbl_e1accdaf-e3bd-4260-a715-737e35a214c1/pull/0.log" Jul 10 23:04:10.543568 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.543480 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd_73e7fae0-1ba2-41e4-a679-85df4e8af497/extract/0.log" Jul 10 23:04:10.570721 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.570685 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd_73e7fae0-1ba2-41e4-a679-85df4e8af497/util/0.log" Jul 10 23:04:10.596671 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.596636 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nvzsd_73e7fae0-1ba2-41e4-a679-85df4e8af497/pull/0.log" Jul 10 23:04:10.905447 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.905282 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2hpnt_f7e899dc-a39f-418e-8ebe-e6fbb9967c09/manager/0.log" Jul 10 23:04:10.932860 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.932823 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-d9pwm_ad0186f0-e17b-4677-85a6-feb5853a3587/manager/0.log" Jul 10 23:04:10.961186 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:10.961141 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zxzzm_b6436dd9-f13f-4e4b-abf0-9fa23884e7ea/kuadrant-console-plugin/0.log" Jul 10 23:04:11.012468 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:11.012431 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zrvs8_99836a7c-8438-49b1-80f6-a34a30e2aa2b/registry-server/0.log" Jul 10 23:04:11.234958 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:11.234889 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-pzp7b_2ffca01f-3f3a-496c-93ac-0d46d9f27915/manager/0.log" Jul 10 23:04:13.320484 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:13.320392 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-68wm9_bc95ce3a-45f8-4017-bd90-510e022d2feb/node-exporter/0.log" Jul 10 23:04:13.343415 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:13.343376 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-68wm9_bc95ce3a-45f8-4017-bd90-510e022d2feb/kube-rbac-proxy/0.log" Jul 10 23:04:13.366708 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:13.366677 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-68wm9_bc95ce3a-45f8-4017-bd90-510e022d2feb/init-textfile/0.log" Jul 10 23:04:16.690475 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:16.690437 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-757cbd9596-qs97t_97cbba24-8d97-4bd2-9a3d-58e79ba08b92/console/0.log" Jul 10 23:04:16.939895 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:16.939853 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8"] Jul 10 23:04:16.945157 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:16.945086 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:16.954625 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:16.954596 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8"] Jul 10 23:04:17.100760 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.100714 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt9x6\" (UniqueName: \"kubernetes.io/projected/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-kube-api-access-jt9x6\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.100983 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.100827 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-proc\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.100983 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.100889 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-podres\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.100983 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.100921 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-lib-modules\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.100983 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.100957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-sys\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.202423 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.202341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt9x6\" (UniqueName: \"kubernetes.io/projected/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-kube-api-access-jt9x6\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.202582 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.202440 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-proc\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.202582 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.202483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-podres\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.202582 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.202501 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-lib-modules\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.202582 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.202534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-sys\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.202759 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.202599 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-proc\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.202759 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.202638 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-sys\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.202759 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.202644 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-podres\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.202759 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.202649 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-lib-modules\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.211757 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.211729 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt9x6\" (UniqueName: \"kubernetes.io/projected/7fba627a-526a-4ec4-b96d-bf7cde6d4a62-kube-api-access-jt9x6\") pod \"perf-node-gather-daemonset-c29d8\" (UID: \"7fba627a-526a-4ec4-b96d-bf7cde6d4a62\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.257773 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.257742 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.436307 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.436273 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8"] Jul 10 23:04:17.438523 ip-10-0-136-68 kubenswrapper[2565]: W0710 23:04:17.438483 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7fba627a_526a_4ec4_b96d_bf7cde6d4a62.slice/crio-b1dba0c5dbf9dc127b1818213740ebd43db61eba98d60cc71ab4d27ef893d53b WatchSource:0}: Error finding container b1dba0c5dbf9dc127b1818213740ebd43db61eba98d60cc71ab4d27ef893d53b: Status 404 returned error can't find the container with id b1dba0c5dbf9dc127b1818213740ebd43db61eba98d60cc71ab4d27ef893d53b Jul 10 23:04:17.618091 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.618046 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" event={"ID":"7fba627a-526a-4ec4-b96d-bf7cde6d4a62","Type":"ContainerStarted","Data":"4d448a0fd000dfeb250283be1615e58fcd3958a7cde704879aec102a9f96658a"} Jul 10 23:04:17.618283 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.618104 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" event={"ID":"7fba627a-526a-4ec4-b96d-bf7cde6d4a62","Type":"ContainerStarted","Data":"b1dba0c5dbf9dc127b1818213740ebd43db61eba98d60cc71ab4d27ef893d53b"} Jul 10 23:04:17.618283 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.618251 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:17.636291 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:17.636224 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" podStartSLOduration=1.636202135 podStartE2EDuration="1.636202135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 23:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:04:17.633004744 +0000 UTC m=+2521.465889129" watchObservedRunningTime="2026-07-10 23:04:17.636202135 +0000 UTC m=+2521.469086519" Jul 10 23:04:18.557967 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:18.557937 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vmg2n_53a16bcf-6853-40fb-99ba-a55039d01d3d/dns/0.log" Jul 10 23:04:18.579696 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:18.579663 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vmg2n_53a16bcf-6853-40fb-99ba-a55039d01d3d/kube-rbac-proxy/0.log" Jul 10 23:04:18.627438 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:18.627385 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g96gd_9f0fd957-b6e9-4ba3-9d5d-8ed753694c91/dns-node-resolver/0.log" Jul 10 23:04:19.204635 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:19.204606 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5zb6h_ee84dbb8-64c7-46d8-840a-94cc40156e29/node-ca/0.log" Jul 10 23:04:20.201453 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:20.201423 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678spvxj_897507dc-df36-48fc-a691-a9e28fec3394/istio-proxy/0.log" Jul 10 23:04:20.306422 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:20.306393 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fc969b7d8-ps4x6_27a335eb-0ac8-49e3-8ea1-31491736893b/kube-auth-proxy/0.log" Jul 10 23:04:20.365000 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:20.364963 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sgnbx_a688576f-a240-451c-84a0-31ef6c45f6a9/istio-proxy/0.log" Jul 10 23:04:20.407550 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:20.407515 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sgnbx_a688576f-a240-451c-84a0-31ef6c45f6a9/istio-proxy/1.log" Jul 10 23:04:20.444385 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:20.444351 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-xqdf8_3ff97a1d-ad63-41f6-8768-b90dcfb72795/payload-pre-processing/0.log" Jul 10 23:04:20.478819 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:20.478772 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-h7xq9_0f9a29f3-8817-4929-bc2c-7d8be1c699b9/payload-processing/0.log" Jul 10 23:04:21.195048 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:21.195014 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-84fbb_38a4df0c-b54e-4152-868e-f71e9f3e7e32/serve-healthcheck-canary/0.log" Jul 10 23:04:21.716181 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:21.716154 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dlrg8_e70e246e-e8b7-4709-aef7-1f23e6aecf7a/kube-rbac-proxy/0.log" Jul 10 23:04:21.737896 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:21.737838 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dlrg8_e70e246e-e8b7-4709-aef7-1f23e6aecf7a/exporter/0.log" Jul 10 23:04:21.760156 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:21.760127 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dlrg8_e70e246e-e8b7-4709-aef7-1f23e6aecf7a/extractor/0.log" Jul 10 23:04:23.639641 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:23.639613 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-c29d8" Jul 10 23:04:24.027856 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:24.027813 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bj7fm_30f4ad53-fa0f-4a7b-af87-20b48ae7783e/manager/0.log" Jul 10 23:04:24.220281 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:24.220242 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-795cbfd498-wtkgt_ba17e396-b47f-494b-9efd-057aa65a379a/manager/0.log" Jul 10 23:04:24.249772 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:24.249745 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zvqhz_20e39149-1c1b-44d5-939d-88ab9382fec7/server/0.log" Jul 10 23:04:24.317994 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:24.317902 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pgqgb_8c0bb4d2-6c19-49bd-b0cf-1f49619cfb0b/manager/0.log" Jul 10 23:04:24.465745 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:24.465708 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58f8cd5b6f-t88g8_ecd095a4-0f72-40d2-98c1-1e6f7421a3ea/manager/0.log" Jul 10 23:04:24.485840 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:24.485818 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58f8cd5b6f-t88g8_ecd095a4-0f72-40d2-98c1-1e6f7421a3ea/copy-manifests/0.log" Jul 10 23:04:25.752980 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:25.752941 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-4f8x9_d6a6b966-a9e5-49f7-bb52-44f0e861bc07/openshift-lws-operator/0.log" Jul 10 23:04:26.891931 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:26.891903 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678spvxj_897507dc-df36-48fc-a691-a9e28fec3394/istio-proxy/0.log" Jul 10 23:04:26.972510 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:26.972481 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fc969b7d8-ps4x6_27a335eb-0ac8-49e3-8ea1-31491736893b/kube-auth-proxy/0.log" Jul 10 23:04:27.010590 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:27.010561 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sgnbx_a688576f-a240-451c-84a0-31ef6c45f6a9/istio-proxy/0.log" Jul 10 23:04:27.052266 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:27.052224 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sgnbx_a688576f-a240-451c-84a0-31ef6c45f6a9/istio-proxy/1.log" Jul 10 23:04:27.076082 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:27.076045 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-xqdf8_3ff97a1d-ad63-41f6-8768-b90dcfb72795/payload-pre-processing/0.log" Jul 10 23:04:27.099576 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:27.099548 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-h7xq9_0f9a29f3-8817-4929-bc2c-7d8be1c699b9/payload-processing/0.log" Jul 10 23:04:28.289233 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:28.289196 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678spvxj_897507dc-df36-48fc-a691-a9e28fec3394/istio-proxy/0.log" Jul 10 23:04:28.389454 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:28.389346 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fc969b7d8-ps4x6_27a335eb-0ac8-49e3-8ea1-31491736893b/kube-auth-proxy/0.log" Jul 10 23:04:28.454202 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:28.454169 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sgnbx_a688576f-a240-451c-84a0-31ef6c45f6a9/istio-proxy/0.log" Jul 10 23:04:28.493856 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:28.493822 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sgnbx_a688576f-a240-451c-84a0-31ef6c45f6a9/istio-proxy/1.log" Jul 10 23:04:28.528651 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:28.528623 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-xqdf8_3ff97a1d-ad63-41f6-8768-b90dcfb72795/payload-pre-processing/0.log" Jul 10 23:04:28.562734 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:28.562694 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-h7xq9_0f9a29f3-8817-4929-bc2c-7d8be1c699b9/payload-processing/0.log" Jul 10 23:04:31.365504 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:31.365469 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-wt2nr_24be55ba-045b-4f39-8af7-528f580ee48c/migrator/0.log" Jul 10 23:04:31.386094 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:31.386064 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-wt2nr_24be55ba-045b-4f39-8af7-528f580ee48c/graceful-termination/0.log" Jul 10 23:04:32.917392 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:32.917364 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ln9m5_1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc/kube-multus-additional-cni-plugins/0.log" Jul 10 23:04:32.938742 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:32.938719 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ln9m5_1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc/egress-router-binary-copy/0.log" Jul 10 23:04:32.959097 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:32.959071 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ln9m5_1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc/cni-plugins/0.log" Jul 10 23:04:32.979505 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:32.979481 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ln9m5_1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc/bond-cni-plugin/0.log" Jul 10 23:04:33.001588 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:33.001560 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ln9m5_1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc/routeoverride-cni/0.log" Jul 10 23:04:33.021492 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:33.021464 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ln9m5_1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc/whereabouts-cni-bincopy/0.log" Jul 10 23:04:33.041901 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:33.041870 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ln9m5_1d7db564-4ffe-4ae3-85ba-4fa345d6d2bc/whereabouts-cni/0.log" Jul 10 23:04:33.229358 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:33.229323 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dwzdh_7a79673b-9490-4771-a6ff-5dd650bf6c28/kube-multus/0.log" Jul 10 23:04:33.345373 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:33.345330 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nwdcr_897ef813-52ab-407d-8429-2656360c5160/network-metrics-daemon/0.log" Jul 10 23:04:33.363431 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:33.363404 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nwdcr_897ef813-52ab-407d-8429-2656360c5160/kube-rbac-proxy/0.log" Jul 10 23:04:34.202356 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:34.202273 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-controller/0.log" Jul 10 23:04:34.218273 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:34.218248 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/0.log" Jul 10 23:04:34.218450 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:34.218351 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovn-acl-logging/1.log" Jul 10 23:04:34.236556 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:34.236534 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/kube-rbac-proxy-node/0.log" Jul 10 23:04:34.257954 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:34.257922 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/kube-rbac-proxy-ovn-metrics/0.log" Jul 10 23:04:34.280357 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:34.280326 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/northd/0.log" Jul 10 23:04:34.302340 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:34.302304 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/nbdb/0.log" Jul 10 23:04:34.324318 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:34.324281 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/sbdb/0.log" Jul 10 23:04:34.385061 ip-10-0-136-68 kubenswrapper[2565]: I0710 23:04:34.385030 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwjng_4c95ccb2-4649-40f6-ae4b-9cdd312797ec/ovnkube-controller/0.log"